Phone Mirror & Android Auto

Overview

OCTAVE has two ways of putting a connected Android phone on the dash:

• Phone Mirror -- OCTAVE is its own scrcpy-protocol client. It pushes OCTAVE's forked scrcpy server to the phone over adb, decodes the H.264 stream itself, plays the phone's audio, and injects real multitouch over the control socket. The server jar is bundled; adb and a decoder must also be available (see prerequisites below). It is desktop-only (mobile builds register a stub).
• Android Auto -- real Android Auto from the phone, with two backends picked by the androidAutoBackend setting:
  • Built-in ("native", beta) -- OCTAVE is the car's head unit. It speaks the Android Auto Protocol (AAP) itself to the phone's Android Auto head unit server through an adb port forward (phone port 5277), decodes the video and plays the audio. No Google software is needed on the OCTAVE machine. See Built-in Android Auto.
  • Google DHU ("dhu", the default) -- runs Google's Desktop Head Unit as a separate process and is meant to embed its window. In practice that window is not shown inside OCTAVE; see Google DHU backend: what actually works.

The paths share adb, found once by PhoneMirrorManager::findAdb() and reused by Android Auto, and the built-in Android Auto receiver picks the phone the same way mirroring does. Phone Mirror and built-in Android Auto both decode H.264 with the shared H264Decoder (src/util/h264decoder.{h,cpp}) and push frames into a QVideoSink bound to a QML VideoOutput; the DHU backend launches the DHU in its own window and registers a QQuickImageProvider (image://dhuframe/) that no QML currently reads.

Phone Mirror

Phone Mirror shows an Android phone inside OCTAVE and lets the user drive it from the dash with real multitouch. OCTAVE is a scrcpy-protocol client: it pushes the bundled phone server jar (tools/phone-server/octave-phone-server) to /data/local/tmp/octave-phone-server.jar with adb, starts it with app_process, decodes the H.264 stream and injects input over the control socket. The server is OCTAVE's fork of the scrcpy server (phone_server/, Apache-2.0, forked at v3.3.4, Java package org.octave.phoneserver, version string 3.3.4-octave). It needs shell privileges, which only an adb-launched process gets, so it cannot be an installed app. There is no external scrcpy binary, v4l2loopback or ffmpeg subprocess in the mirroring path. Touch uses the control socket; wakePhone() still uses adb shell input keyevent 224 as a wake-up fallback.

Logging: categories octave.phonemirror (manager) and octave.phonemirror.client. Server log lines are relayed at INFO (WARN/ERROR lines at warning level) with a server: prefix.

Measured on the Orange Pi 5 Plus with a Galaxy S22 Ultra (Android 16), from the click on the button: handshake at +0.51 s, first decoded frame at +0.65–0.83 s regardless of how long the phone has been idle, and a tap visible in the stream after ~0.4 s. Numbers and history: docs/PHONE_MIRROR_NATIVE_PLAN.md.

Tests (QtTest, tests/cpp/): tst_scrcpyclient runs the client against a fake scrcpy 3.x server (handshake, config and key packets, a mid-stream resolution change, a corrupt packet, a garbage header, early exits), with tests/cpp/fake-adb.sh standing in for adb and the H.264 clips in tests/fixtures/h264/; tst_phonemirror covers the manager (adb parsing, serial choice, Wi-Fi messages, QR pairing, the media hand-off rules, notices, the encoder fallback and the async start flow); tst_phonemirrorview drives PhoneMirrorView.qml against a fake manager; tst_adaptivebitrate feeds simulated packet streams to the adaptive bit-rate logic.

PhoneMirrorManager

Owns the single session (one phone at a time), the adb probes the UI needs, and the settings that shape a session. Mobile builds compile a reduced compatibility stub that reports nativeAvailable = false and "Phone mirroring is desktop-only".

Properties

Signals

Slots

Session lifecycle

After manager preflight, session setup and video decoding run on the client's worker thread (a std::thread named scrcpy-client).

1. Preflight: nativeAvailable on the GUI thread; then, on the probe thread (so a dead Wi-Fi link or an adb restart never freezes the dash), adb devices, the phone to use (the one picked with selectDevice(), else USB first), getprop ro.serialno (the phone's identity, whatever the transport), a stale-server kill (adb shell pkill -f org.octave.phoneserver) unless reattaching to this same phone's persisted session, and getprop ro.build.version.sdk. The result is handed back to the GUI thread, which drops it if stopScrcpy() ran meanwhile (a generation counter). ScrcpyClient::bundledServerJar() extracts the embedded jar to <temp>/octave-phone-server-3.3.4-octave (reused when the size matches).
2. Push the jar to /data/local/tmp/octave-phone-server.jar -- every session, because cleanup=true starts a CleanUp helper that unlinks the pushed jar as soon as the server is up.
3. Tunnel: pick a random 31-bit scid (QRandomGenerator::system()), formatted as 8 hex digits. Reap stale forwards first (adb forward --list, remove every localabstract:scrcpy_* entry for this serial -- forwards live in the adb server, so a hard-killed OCTAVE strands one listening port per session), then adb forward tcp:PORT localabstract:scrcpy_SCID with an ephemeral local port (OCTAVE_SCRCPY_PORT overrides it for the fake-server tests in tst_scrcpyclient; it falls back to 27183 if the probe listen fails).
4. Launch:

   adb -s SERIAL shell "CLASSPATH=/data/local/tmp/octave-phone-server.jar nohup app_process / org.octave.phoneserver.Server 3.3.4-octave \
     scid=SCID tunnel_forward=true video=true audio=true|false control=true clipboard_autosync=false \
     video_codec=h264 audio_codec=raw max_size=0 video_bit_rate=8000000 max_fps=60 stay_awake=true cleanup=true \
     send_device_meta=true send_frame_meta=true send_codec_meta=true send_dummy_byte=true log_level=info \
     octave_persist_ms=60000|0 [video_codec_options=...] [new_display=WxH display_ime_policy=local] 2>&1 & wait"

   The CLASSPATH= assignment must come before nohup (an assignment after it is exec'd as a literal command; commit bba3de6). nohup ... & wait keeps the adb shell relaying stdout for as long as the server lives, while the server survives adbd killing that shell on a USB dropout (it is reparented to init). octave_persist_ms is 60000 for a virtual-display session and 0 (exit like scrcpy) when mirroring the phone screen. On Linux the adb child gets PR_SET_PDEATHSIG=SIGTERM (setChildProcessModifier) so it dies with OCTAVE. The child's merged stdout/stderr is the server log; a New display: ... (id=N) line yields the virtual display id used by pressHome(). video_codec_options defaults to priority:int=0,latency:int=1,vendor.qti-ext-enc-low-latency.enable:int=1 (ScrcpyClient::kVideoCodecOptions): realtime priority, no frame lookahead and Qualcomm's low-latency mode, which took the lag out of dragging on an S26 Ultra. Encoders ignore keys they do not know. The OCTAVE_VIDEO_CODEC_OPTIONS environment variable replaces the list (empty sends none). On Windows ScrcpyClient::hideConsoleWindow() sets CREATE_NO_WINDOW on every adb / server QProcess (probes, stale-server kill, pressHome, wakePhone, the session's adb child) so no console window flashes up.
5. Connect: with tunnel_forward the local TCP connect succeeds before the server listens and then EOFs, so connect every 100 ms until the dummy byte actually arrives; that socket is the video socket. Deadline 20 s from the click (4 s when attaching); if the server process exits meanwhile the failure is "scrcpy server exited during startup (see log)". Then open the audio socket (only if audio is on) and the control socket (both with TCP_NODELAY). The server sends device and codec meta only once every socket it expects is connected (DesktopConnection.open accepts them in order), so reading before opening control would deadlock. A connection that received no dummy byte is closed, otherwise the server would later hand it out as the audio or control socket.
6. Handshake on the video socket: 64-byte NUL-padded device name, then codec id (u32 FourCC h264), width and height (u32 each, big-endian). frameSizeChanged and connected fire; the manager emits scrcpyStarted(1) and, 500 ms later, sends the keeper policy (the server powers the device on asynchronously at start).
7. Stream: the worker thread demuxes and decodes video until stop or EOF; a scrcpy-devmsg thread reads device messages on the control socket; a scrcpy-audio thread reads PCM. Control messages are written to the control socket from any thread under a send mutex, straight to the raw descriptor with ::send().
8. Teardown: close the sockets, terminate the adb child (2 s, then kill), adb forward --remove tcp:PORT (also repeated from stop() so a clean exit never strands a port). stop() never waits out a slow step: adb calls and the connect wait poll m_stopping every 100-200 ms (cleanup calls excepted). The audio socket belongs to the audio thread; the session only shuts its descriptor down, under a mutex the audio thread takes before closing it. The server destroys the virtual display itself (cleanup=true) unless it is persisting. A stream that ends without a requested stop surfaces as "Phone disconnected. Reconnect the USB cable.". src/main.cpp routes SIGTERM/SIGINT through the normal quit path (a self-pipe socketpair handler) so launcher scripts and systemd get this cleanup too.

Protocol byte layouts

All integers big-endian. Verified against phone_server/src/main/java/org/octave/phoneserver/ (device/DesktopConnection, device/Streamer, control/ControlMessageReader, control/DeviceMessageWriter) and src/phone_mirror/scrcpyclient.cpp; the same table lives in docs/PHONE_MIRROR_NATIVE_PLAN.md.

Sockets (in accept order, all on localabstract:scrcpy_<scid>)

Stream packets (video and audio, send_frame_meta=true)

[u64 pts_and_flags][u32 size][size bytes]
  bit 63 = config packet (SPS/PPS for H.264; pts invalid)   // Streamer.PACKET_FLAG_CONFIG
  bit 62 = key frame                                          // Streamer.PACKET_FLAG_KEY_FRAME
  video data: Annex-B H.264; audio data: PCM s16le 48 kHz stereo (AudioConfig, up to 4096 B per read)

The client treats a size above 32 MiB (video) or 1 MiB (audio) as a desynced stream: video fails the session ("video stream out of sync"), audio stops and video continues.

Control messages (client → server), first byte = type

Device messages (server → client on the control socket), first byte = type

Coordinates are relative to the mirrored display: the server routes injection to the display it captures, so a virtual display needs no display-id routing on the touch path.

Video decode and frame delivery

A config packet (SPS/PPS) is not decoded on its own; it is prepended to the next packet, as the scrcpy client does. The client decodes with libavcodec through H264Decoder (src/util/h264decoder.{h,cpp}, shared with the built-in Android Auto receiver; it owns the codec context, the SPS/PPS prepend and the frame copy, while the black-frame hold and size tracking stay in ScrcpyClient's frame callback) (avcodec_find_decoder(AV_CODEC_ID_H264), slice threading only, set before avcodec_open2, packets padded with AV_INPUT_BUFFER_PADDING_SIZE). Frame threading is avoided because it delays every picture by one frame per thread. It accepts YUV420P/YUVJ420P (other formats are dropped) and copies the three planes row by row into a QVideoFrame(Format_YUV420P) -- no RGB conversion, the GPU does the colour conversion in VideoOutput. The frame is stored as "latest" under a mutex and frameReady is emitted; a queued slot on the GUI thread pulls the latest frame and calls QVideoSink::setVideoFrame(), so a slow GUI thread coalesces rather than queues frames. The client never clears the sink: the last frame stays on the VideoOutput across a reconnect.

Why libavcodec and not QMediaPlayer::setSourceDevice(): Qt's FFmpeg backend refuses a raw Annex-B QIODevice (FormatError "Could not open file" after 1.5 MB fed, Phase 0 spike). OCTAVE therefore links libavcodec/libavutil when CMake finds them (OCTAVE_HAVE_FFMPEG, pkg-config on Linux/macOS, vcpkg on Windows; without them the build still succeeds and nativeAvailable is false). The native-client plan measured libavcodec's software H.264 decoder at 1280x800 at 12–15 % of one RK3588 core; this is hardware-specific.

Black-frame holding

A dozing phone streams pure black frames (status bar still drawn) until it is woken, and the first composite of a reattached stream is black plus a strip of status icons. Rather than flash the dash black, the client decides per decoded frame whether to hand it to the sink:

• Test: maximum luma over every other pixel in both directions of the Y plane (stride 2), early exit on the first sample above 20. Point grids (8x8, then 24x24) were tried first and missed a 2 px icon stroke, so a dark app with small bright UI was held as if the phone were dozing (commit 7090531). ~256k byte reads on 1280x800, under 1 ms. Frames narrower or shorter than 24 px are never "black".
• Lenient mode (attached stream, first 3 s of decode): the frame is black if fewer than 5 % of samples are bright -- a solid status strip is 3–4 % of samples (commit cfc1d4c). Normal viewing keeps the strict test so a dark app with a bright toolbar is never held.
• Hold rules: while the keeper reports the phone asleep, every frame is held (the doze fade is a uniformly dimmed composite, neither black nor content; commit 525882f). Otherwise a run of black frames is held for 3 s (kBlackHoldMs); a black run that outlives that is real content and is shown. The very first frame of a fresh session is always shown (the test only applies once a frame has been delivered, or when attached).

Audio and ducking

Phone audio always plays through OCTAVE while mirroring (the scrcpyAudioEnabled toggle was removed): the server is started with audio=true audio_codec=raw, and the stream follows OCTAVE's volume (setVolume) and mute (setMuted, wired from VolumeController.mutedChanged); the server's default source is AudioSource.OUTPUT, i.e. REMOTE_SUBMIX, which captures every app and mutes playback on the handset. Raw PCM means no decoder on our side. The audio socket is read on its own thread (scrcpy-audio) so video never waits on it:

1. Each chunk's peak before gain is measured; a peak of 164 or more (about −46 dBFS; a silent remote submix is exact zeros, so this only rejects dither) flags "sound seen".
2. Gain (setting scrcpyAudioGain, default 2.0 = +6 dB, clamped 0.25..8) multiplies the samples through a tanh soft limiter so a hot source cannot clip harshly. Phones capture at their own media level, measured ~10 dB under local music on the Pi; the phone's own volume is deliberately never touched (an earlier attempt used adb shell media volume, which does not exist on the test handset and would leave the phone at maximum after a crash; commit 19c3b10).
3. Chunks are queued with a cap of 48000 bytes (~250 ms); when the GUI thread falls behind the oldest are dropped so audio stays close to the picture. audioReady wakes a queued slot on the GUI thread.
4. On the GUI thread a QAudioSink (48 kHz, stereo, Int16, default output device, ~200 ms device buffer) is created lazily; chunks are written only while bytesFree() allows, never blocking. setVolume() from VolumeController sets the sink volume, so the knob and mute apply. If the default device is missing or refuses the format, a warning is logged and audio is silently skipped.
5. The same GUI slot turns "sound seen" into audioPlaying = true and (re)arms a single-shot 1.5 s release timer; silence longer than that drops it. Long enough to bridge the gaps between navigation sentences, short enough that music comes back promptly after a prompt.

Ducking (settings scrcpyAudioDuckEnabled, default on; scrcpyAudioDuckLevel, default 0.1 = −20 dB): the manager computes duckingFactor = duckLevel while audioPlaying && duckEnabled, else 1.0, and emits duckingChanged on change. main.cpp connects it to MediaManager.setDucking(), which ramps the local output (80 ms attack, 500 ms release) independently of the user's volume and mute. Expect lip-sync to trail the picture slightly. Validated on the Pi by measurement (+6 dB = x1.89 RMS, live slider, clean teardown with zero underruns); whether −20 dB is enough against a loudness-mastered MP3 in the car is still an open question in the plan doc.

Phone screen, sleep and the MirrorKeeper

Measured on the S22 Ultra / Android 16: the virtual display requests VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP but lands in power group 0, so a power press dozes it with the phone -- app content stops compositing (black frames) and injected touch is silently dropped, with nothing on the wire to notice. The only way to keep mirroring is to wake the phone again. Doing that from the dash (an adb poll every second, then a keyevent) left ~1 s of black and a visible panel relight, so the logic moved into the phone: phone_server/.../control/MirrorKeeper.java (commit 67809d1). OCTAVE only sets the policy, can take the phone back, and shows the state; it no longer polls adb to maintain the panel policy. Explicit wake and virtual-display Home actions still launch adb commands.

Policy: OCTAVE_SET_KEEPER(enabled=1, panelDark=scrcpyPhoneScreenOff, graceMs=8000), sent 500 ms after the handshake and again whenever the setting changes (the server's own default grace is 5000 ms if a client never sets one). The keeper is a process-wide singleton on the phone: Controller attaches it on the first OCTAVE_SET_KEEPER, swaps the message sender per client session, and its in-use / panel state survives a reconnect.

Sensing: a DisplayListener on display 0 plus a 250 ms poll on its own HandlerThread, reading PowerManager.isScreenOn(0) (interactive) and IWindowManager.isKeyguardLocked() through the server's wrappers. The panel is blanked or lit with Device.setDisplayPower(0, on) (SurfaceControl power mode -- the phone stays awake, stream and touch keep working) and CleanUp is told to restore it when the server process ends. Waking uses KEYCODE_WAKEUP (224, wakes and never toggles off), rate-limited to one per 500 ms.

Idle timeout: a woken lock screen overrides the phone's screen timeout with a few seconds (mUserActivityTimeoutOverrideFromWindowManager=5000 on a Galaxy S-series), and stay_awake only holds while the phone is charging. Left alone, a cradled locked phone therefore dozed every ~5 s; each doze blanked the mirror for a moment and looked like a power press, so the panel was relit. While enabled, awake and not in use, the keeper calls IPowerManager.userActivity() every 1.5 s (the shell holds DEVICE_POWER), so the only dozes left are real power presses.

Decisions (what a power press meant):

• Start: an awake, unlocked phone is in the user's hand (in use); an awake, locked one is in the cradle and gets the panel policy; an asleep one is woken.
• Press while locked (phone in the cradle): wake it, then force the panel on once interactive (the wake alone does not reliably relight a SurfaceControl-dark panel; commit 2cdd28e). The forced relight only runs while panelDark is on: setting the power mode bypasses the brightness controller and wakes a Galaxy S22 dim, so a phone the keeper never darkened is left alone. Leave it lit for the 8 s grace window. If the keyguard is dismissed in that window (or at any time), or the phone turns out to be unlocked already (a lock-after delay had not run), or the button is pressed again within grace + 2 s (nothing visible happened for them the first time, so they are trying to get in), the phone is in use: panel on, locks not undone. If the grace passes still locked, the press meant "screen off" and the panel is blanked.
• In use: the mirror keeps working (DeX on the virtual display is independent of the panel). It ends when the keyguard re-engages (lock shortcut or lock-after delay: grace, then blank) or on OCTAVE_TAKE_BACK.
• Press while in use: the user is putting the phone down -- no grace, the panel-off goes out before the wake and again the instant the phone is interactive, so it never relights.
• In-use is edge-triggered on the keyguard transition (locked → unlocked enters, unlocked → locked leaves); a level check raced Samsung's 5 s lock-after delay and undid resumeMirroring() on the next poll (commit e1fa1c4).

Every change is reported as PHONE_STATE(asleep, inUse, panelDark). The manager maps it to phoneAsleep / phoneInUse, tell the client to hold every frame while asleep, and reset both on disconnect. PhoneMirrorView dims the held frame with a "Phone is asleep -- Waking it up... tap to retry" overlay only if the phone is still asleep after 1.5 s; a normal press resolves in about a second and dimming the dash for that read as a flash (commit a4aa9a6). Tapping the overlay calls wakePhone().

Phone media: OCTAVE's controls drive the phone

Music playing on the phone (any app with a media session: Spotify, YouTube Music, podcasts...) shows up in OCTAVE's own now-playing UI and is driven by its normal controls: the bottom bar, the Media Room, the home screen card, the dashboard NowPlayingWidget / MediaControlsWidget, gesture swipes and Album Art Capture theming. These controls and metadata use the control socket, independently of whether the phone successfully supplies audio.

Phone side: phone_server/.../control/PhoneMedia.java, created by Controller on the first OCTAVE_MEDIA and stopped with the client session. It lists sessions through the media_session binder (ISessionManager.getSessions(null, 0[, null]): the third argument, a package filter, appeared in Android 17; the shell holds MEDIA_CONTENT_CONTROL), follows the one that is playing or else the highest priority one (re-picked every second), and registers a MediaController.Callback on it, so changes are reported as they happen. Transport goes through MediaController.TransportControls; with no session, play falls back to KEYCODE_MEDIA_PLAY. Art (album art, art or display icon) is scaled to 400 px and sent as JPEG only when it changes (identity: size and a 4×4 pixel sample); a "no art" after a track change is held back 1 s because apps send the title first and the art a moment later.

Who has the controls (mediaActive): the phone is the media source for the whole mirroring session. It takes the controls with the session's first media report (sent right after WATCH, even with no media app), keeps them while paused or with no media app, and loses them when the session ends or releaseMedia() is called; after that it takes them back only when it starts playing again. Pressing play with no media app sends KEYCODE_MEDIA_PLAY, which restarts the last media app. NowPlaying holds the hand-offs: when the phone takes over, local playback pauses; a local track starting calls releaseMedia(true) (the phone pauses). Frontends read the live source through nowPlaying, not these properties. With no media app the views show "Phone" / "Nothing playing". The Media Room shows a "Phone" source pill meanwhile; tapping it (nowPlaying.useLocal()) hands the controls back to local media and pauses the phone. The same rules with Android Auto: Which source plays the music. Shuffle is dimmed (it belongs to the phone app) and there are no side cards.

Queue: PhoneMedia follows the selected app’s MediaController queue and active queue ID, sending up to 500 entries. Queue artwork is fetched separately, scaled to 160 px and sent once per URI per session. The manager stores it under <app data>/temp/phone_queue_art/ and batches artwork notifications for 150 ms. MediaPlayer.qml shows this queue while the phone is active and a queue is available; selecting an entry calls mediaSkipToQueueItem(id). The available tracks and skip behavior depend on what the phone app publishes. This is its active queue, not a browsable account library.

Waveform: the client emits audioPlayed(pcm, bufferedMs) as audio is handed to the sink. main.cpp routes it to AudioAnalyzer.feed_live_pcm(); main.cpp also switches it to live analysis while nowPlaying's source is the phone. Without captured phone audio, the controls can still work but there is no live PCM waveform.

QML reads what is playing from nowPlaying, which follows the phone while mediaActive and local media otherwise. Position is not streamed by the phone: nowPlaying extrapolates mediaPosition() every 500 ms while the phone plays (one clock for every view). Every view uses it; phone-only features still read phoneMirrorManager: the queue (MediaPlayer.qml, shown while nowPlaying.source is the phone, following source changes while the page is open) and Media Room's "Phone" pill (releaseMedia(true)).

Persistent sessions and USB drops

The phone re-enumerates on the bus now and then (picked up, put down, a nudged cable). adbd then kills the server's shell; with the stock scrcpy design the server exits, the virtual display goes with it (VIRTUAL_DISPLAY_FLAG_DESTROY_CONTENT_ON_REMOVAL) and the user lands on a fresh DeX home screen. Two layers keep what they had open (commit 2992d67):

Server (option octave_persist_ms): Server.scrcpy runs a session loop on a sessions thread while the main looper serves handlers for the whole process. One NewDisplayCapture marked persistent is shared across sessions -- its release() only detaches the encoder surface (virtualDisplay.setSurface(null)), the real destroy() runs at process exit. Each client connection gets a fresh Controller, encoder and streamer and re-sends device meta. After a client drops, DesktopConnection.open(..., acceptTimeoutMs) re-listens on the same localabstract:scrcpy_<scid> for up to octave_persist_ms; because closing a LocalServerSocket does not reliably unblock accept() on Android, a watchdog thread sets a flag and pokes the socket with a connection of its own (commit 0a26f8e). No reconnect within the window: "No client reconnected within N ms; exiting", the display is destroyed and MirrorKeeper.shutdown() runs.

Client: on a mid-stream disconnect of a ready virtual-display session the manager remembers the scid and a deadline of now + 60 s − 5 s. The next startScrcpy() inside that window attaches, but only to the same phone: the session's ro.serialno is kept with the scid, and a different phone picked inside the window starts fresh instead of spending the 4 s attach deadline on a session that isn't there (an unknown serial keeps the time-only rule). Attaching means no push, no pkill, re-forward the same scid, connect with a 4 s deadline, and follow the server log with adb logcat -v raw -T 1 -s scrcpy:I (its stdout went with the old shell). The reattached stream uses the lenient black test for its first 3 s so the handover never shows black. If the attach fails, the manager starts fresh on the next event-loop turn (which kills the old server). A deliberate stopScrcpy() always kills the server.

Recovery is the manager's (sessionState): a device error (lastErrorKind "device" or "noPhone") after the stream had been up is a quiet reconnect ("reconnecting"): it polls every second (kReconnectPollMs) and restarts the moment adb reports the phone (typically 2 s), keeps reconnecting through failed retries, and moves to "waitingForPhone" after kReconnectGraceMs (10 s). Any other device error goes straight to "waitingForPhone", which polls every 2 s (kWaitPollMs) and restarts when the phone is back; setup and fatal errors are "failed" and never retried. Polling runs only while setViewActive(true) (the mirror page is on screen), so nothing spawns adb in the background. The page holds the last frame while "reconnecting" (a "Reconnecting to phone..." badge after 2 s) and shows the error page otherwise. A dropped session's server persists for 55 s, so a phone back after the 10 s grace still reattaches. Because the keeper is process-wide, in-use / panel state survives the reconnect too.

Mirroring over Wi-Fi

adb carries the whole session over TCP, so the client needs no changes for Wi-Fi: once the phone is listed by adb devices -- as "ip:port" after adb connect, or as an mDNS name (adb-<serial>-<id>._adb-tls-connect._tcp) when adb auto-connects a phone it has paired with -- the client's adb -s <serial> push, forward and shell work exactly as over USB. The manager only gets the phone into that list and keeps it there. ScrcpyClient::isNetworkSerial() tells the two apart (USB serials never contain :).

QR pairing is the wireless setup offered by the UI (Android 11+); a USB session can also enable the legacy TCP connection described below. Typing a pairing code, an IP:port, or scanning for adb services was removed again in favour of it; USB needs no setup and stays as it was.

• Where the code appears: PhoneMirrorView becomes the QR screen whenever no phone is connected and none is remembered (noPhone: the error text starts with "No Android device"); it starts pairing on its own and cancels when the view is left or mirroring starts. With a remembered phone the page says "Waiting for your phone" and offers "Connect a different phone". In Settings → Accessories → Phone Dock the "Connect phone" button opens the same panel in a centred modal. Both use frontend/PhoneQrPairingPanel.qml: the code on a Canvas, black on white with a 4-module quiet zone, sized from the screen height (55 %) so it scans from arm's length, the three steps, the result line and Cancel / "Show QR code".
• Protocol (Android Studio's): startQrPairing() makes a random service name (octave-<8 chars>) and password (12 chars) and encodes WIFI:T:ADB;S:<name>;P:<password>;; (qrPairingPayload(), ECC level M) with Project Nayuki's QR library, vendored as src/phone_mirror/qrcodegen.{hpp,cpp} (MIT, credited in NOTICE). The rows reach QML as qrPairingRows. The phone's Wireless debugging → Pair device with QR code reads it and advertises an _adb-tls-pairing service under that name (if the adb in use is older than platform-tools 30, startQrPairing fails at once with "adb is too old for wireless pairing" instead of waiting); the worker polls adb mdns services every second for up to 2 minutes (kQrPairingTimeoutMs), runs adb pair <ip:port> <password>, then finds the phone's _adb-tls-connect port on the same host over mDNS (polled up to 8 s) and runs adb connect (adb exits 0 on failure, so success is read from the output). Cancel bumps a generation counter the waiting job checks every second; the outcome arrives as wirelessResult.
• Remembering and reconnecting: a successful pairing sets wirelessAddress, which main.cpp saves to scrcpyWirelessAddress. At startup and whenever getDeviceState() finds no usable phone (the view polls it), the manager tries that address in the background at most every 5 s. Wireless debugging picks a new port each time it is switched on, so when the saved address fails it looks for an _adb-tls-connect service on the same host and, if that connects, remembers the new port; adb itself also auto-connects paired phones it sees over mDNS. adb can keep listing the old ip:port as a device for a while after the phone rejoins Wi-Fi on a new port, so a session that fails to start on an ip:port serial runs adb disconnect on it, re-resolves the port over mDNS, connects and starts once more (m_staleRetried, one retry per start). Forget (forgetWireless()) stops a Wi-Fi session, runs adb disconnect and clears the address; adb may still auto-connect a paired phone until OCTAVE is removed from the phone's paired devices.
• USB first: pickSerial() prefers the phone picked in the UI, then a USB serial, then Wi-Fi, so a phone on both links mirrors over USB; every adb call for the session (the SDK getprop, the stale-server pkill, push, forward, shell) passes -s, since a bare adb shell fails with "more than one device". Unplugging the cable mid-session drops into the normal quiet reconnect, which then finds the Wi-Fi serial and carries on (attaching to the persisted virtual display).
• Bit rate: per link from the settings, scrcpyBitRateUsb (default 16 Mbps) and scrcpyBitRateWifi (default 8 Mbps). With scrcpyBitRateAuto (default on) those are the ceiling and the rate adapts; see Adaptive quality.
• Wi-Fi without re-pairing (scrcpyAutoWifi, off by default since 0.9.5: until the phone reboots, its adbd then accepts connections on whatever network it joins): before a USB session the manager reads the phone's Wi-Fi IPv4 from ip -4 -o addr show (a wlan* interface first, then a phone-run hotspot swlan* / ap*; mobile data and VPNs are skipped). If service.adb.tcp.port is not already 5555 it runs adb tcpip 5555, which restarts the phone's adbd, and waits up to 9 s for the USB device to return. It then remembers ip:5555 as wirelessAddress. This runs on the wireless worker; the session starts when it finishes, and a stopScrcpy() in between cancels that start (a generation counter). The legacy TCP mode needs no Wireless debugging toggle, works when the phone runs the hotspot, and survives network changes, but ends when the phone reboots. Its connections still need the phone's RSA authorization, like USB. When the saved ip:5555 fails the existing mDNS fallback still finds a Wireless-debugging port on the same host.
• Messages: a dropped Wi-Fi link reports "Phone disconnected. Check that it is on the same Wi-Fi network and Wireless debugging is on." instead of the USB-cable text; offline / unauthorized network devices and "no device while a Wi-Fi phone is remembered" get Wi-Fi wording. All are classified by PhoneMirrorManager::errorKindFor() (tst_phonemirror::everyMessageHasItsKind pins each message's kind).

When adb mdns services returns no usable services, the manager tries avahi-browse on Linux. This supports distro adb builds without working mDNS; the fallback requires avahi-browse to be installed.

Threading: QR waits and reconnects block on adb, so they run on one worker (a QThreadPool with one thread) and post results back to the GUI thread. One worker serialises them, so a background reconnect never races a pairing; the destructor cancels a QR wait and waits for the worker.

Networking: no internet is involved; adb, the video, audio and touch all stay on the local link. OCTAVE does not create the network. The phone and the OCTAVE machine just have to share one: a home router, or in a car the head unit's own access point (e.g. a 5 GHz hostapd / NetworkManager hotspot) that the phone joins. Android reports that network as "no internet" and keeps using mobile data for its apps, which is what you want. A phone-hosted hotspot does not work with Wireless debugging (Android only enables it while the phone is a Wi-Fi client), but it does work with the legacy TCP mode that scrcpyAutoWifi turns on over USB.

What survives what: USB authorization is kept across plug-ins (Android revokes it after 7 days without a connection unless Developer options → "Disable adb authorization timeout" is on). Wireless debugging is switched off by Android when the phone reboots and, on most phones, when it leaves the network it was enabled on. The legacy TCP mode lasts until the phone reboots; one USB session turns it back on.

Adaptive quality

With scrcpyBitRateAuto on, the client keeps the video bit rate as high as the link carries. The logic is one small pure class, src/phone_mirror/adaptivebitrate.h (covered by tst_adaptivebitrate), fed from the video thread for every media packet.

• Signal: each packet's phone presentation time against its arrival on our steady clock. The difference is the one-way delay plus an unknown clock offset; the smallest delay over the last 10 one-second windows stands in for an idle link, so the excess is queueing. It is measured on arrival, before decoding, so a decoder that cannot keep up (large frames on a small board) backs the rate off just like weak Wi-Fi. A jump over 3 s is taken as a clock step and restarts the baseline.
• Decisions (per one-second window, after a 2 s warm-up; the window's single worst packet is left out so a key frame is not congestion): mean queueing over 150 ms cuts the rate to 70 % (at most every 2 s); under 40 ms for two windows in a row, 5 s after the last cut, and with the stream actually using at least half the rate (a still screen proves nothing) raises it by 10 %, at least 1 Mbps. The rate stays between 2 Mbps (kAdaptiveMinBps) and the link's setting. Changes under 5 % are not sent.
• Start: USB at the ceiling, Wi-Fi at 60 % of it (kAdaptiveWifiStart). The starting rate is sent with the first packet too, since a reattached server may still run an earlier session's rate.
• Phone side: OCTAVE_SET_BIT_RATE → VideoBitRate.set() → MediaCodec.setParameters(PARAMETER_KEY_VIDEO_BITRATE) on the running encoder, no restart and no key frame. Encoders that ignore live changes keep their rate; the cut still helps once the encoder is recreated.
• Visible: currentBitRate (bps, 0 when idle) follows every change; the Phone Dock card shows "Now: … Mbps over USB / Wi-Fi", and each change is logged with its queueing delay.

Touch and navigation

PhoneMirrorView.qml uses a MultiPointTouchArea (mouseEnabled, 1–10 points; mouse arrives as one point) and forwards every point's pressed/updated/released (and canceled, as a release) through injectTouch(pointId, action, relX, relY), mapped through VideoOutput.contentRect so letterboxing is accounted for. A touch that starts in the letterbox (phone-screen mode) is ignored for its whole life, so a thumb resting on the bars can't tap the phone's edge; one that starts on the picture is clamped to 0..1 if it drags off it. Tap, drag, scroll and long-press are verified on hardware; multi-finger gestures are implemented but untested (the test rig has no touchscreen).

Navigation row (◁ ○ ▢ → pressBack / pressHome / pressAppSwitch) sits on the bottom edge of the picture (not the letterbox band, where a translucent button on black is invisible) and is shown only when mirroring the phone's own screen: a virtual display draws system decorations, i.e. Samsung's DeX taskbar with its own back/home/recents exactly where the row would sit. HOME and APP_SWITCH are system keys that Android routes to the default display, so injecting them does nothing on a virtual one (Back is dispatched to the focused window and works); pressHome() therefore runs adb shell am start --display <id> -a android.intent.action.MAIN -c android.intent.category.HOME when a virtual display id is known (via an async QProcess) and falls back to the keycode otherwise. Samsung's recents UI renders empty on a secondary display.

Errors and recovery

A missing prerequisite is reported with getInstallInstructions() naming it and the view shows a "Setup" block. Phone-side failures (unplugged, offline, USB debugging not authorized, Linux USB "no permissions") get the one-line describeDeviceState() text; the error page then asks for a probe (refreshDeviceState()) every 2 s while it is on screen and restarts on deviceStateChanged when the phone is back -- a nudged cable in a vehicle is the normal case. Failures that retrying cannot fix (the video encoder) stop the loop.

Encoder fallback: when the server logs a video encoder error (Capture/encoding error, Video encoding error, Could not create (default) video encoder) and the session drops within 5 s, the manager restarts once with video_encoder=c2.android.avc.encoder (Google's software encoder; OMX.google.h264.encoder before Android 10) and keeps using it for that phone until OCTAVE restarts, with a notice. If the software encoder fails too, the view shows the cause and stops retrying.

The bottom-bar button pops back to an existing PhoneMirrorView (found by objectName) instead of stacking a second one, and the view resumes a session that is already running.

Why not QtMultimedia's Camera? The design before the built-in client streamed scrcpy into a v4l2loopback node and read it with QCamera. Qt snapshots the V4L2 device list on first use per process and never refreshes it, the loopback only appears while a producer streams, and Qt's V4L2 path sheared frames whose width was not a multiple of 64. That knowledge still applies to camera enumeration; the path itself is gone.

The phone server fork and the version invariant

phone_server/ is scrcpy's server/ tree at v3.3.4 with the package renamed to org.octave.phoneserver (LICENSE and NOTICE kept, phone_server/LICENSE.scrcpy). OCTAVE's additions are marked with // OCTAVE comments: control/MirrorKeeper.java, the TYPE_OCTAVE_* control and device messages, Options.octavePersistMs, the session loop in Server.scrcpy, DesktopConnection.open(..., acceptTimeoutMs), NewDisplayCapture.setPersistent()/destroy(), and the WindowManager.isKeyguardLocked wrapper.

Build: phone_server/build_without_gradle.sh (upstream's no-Gradle script) generates BuildConfig.VERSION_NAME = "3.3.4-octave" from SCRCPY_VERSION_NAME, compiles against platforms;android-36 / build-tools;36.0.0 and produces octave-phone-server (a jar of classes.dex). The result is committed at tools/phone-server/octave-phone-server; the binary embeds it as the Qt resource :/phone-server/octave-phone-server (CMake qt_add_resources) so no packaging target carries a loose file.

Version invariant: Options.parse throws "The server version (3.3.4-octave) does not match the client (...)" unless the first argument equals BuildConfig.VERSION_NAME. So SCRCPY_VERSION_NAME in the build script and kServerVersion in scrcpyclient.h must be bumped together. The CI job phone-server rebuilds the jar under JDK 21 on every push and errors if either the fresh or the committed jar lacks the org/octave/phoneserver/Server class or the 3.3.4-octave string; a byte-level difference between the two is reported as a warning only (javac/d8 output can differ across toolchains). Changing the server is a deliberate step: edit phone_server/src, rebuild, commit the jar with the source, bump the version in both places if the protocol changed, and re-verify the byte layouts above against DesktopConnection, ControlMessageReader, DeviceMessageWriter and Streamer.

Bundled adb: scripts/fetch_platform_tools.py downloads Google's platform-tools 37.0.1 (SHA-256 verified), keeps only what adb needs plus Google's NOTICE under tools/platform-tools/<linux|windows|darwin>/ (gitignored). The Windows, macOS and Linux x86_64 CI jobs run it and ship the result next to the binary (platform-tools/ beside the executable, Contents/Resources/platform-tools in the .app, usr/bin/platform-tools in the AppImage). Google publishes x86_64 only, so the aarch64 AppImage and ARM boards fall back to the distro's adb on PATH.

Settings and wiring

src/main.cpp does the wiring: read each setting into the manager at startup, connect the SettingsManager change signals to the live setters, and connect duckingChanged to MediaManager.setDucking. The Accessories settings page shows a status line: "Built-in mirror client ready (scrcpy server 3.3.4-octave, adb: ...)" or the install instructions. Full schema: Settings Reference.

Design decisions

Grounded in the code comments, commit messages and docs/PHONE_MIRROR_NATIVE_PLAN.md.

Why a built-in client instead of the scrcpy binary

The first working path (commit 0f9ecaa) chained the scrcpy desktop binary, a v4l2loopback kernel module, an ffmpeg subprocess reading the loopback node, adb shell input for touch, and on Windows a GDI window grab. Every link was an install step or a failure mode on someone else's box: distro scrcpy versions are years old (Ubuntu 22.04's 1.21 cannot mirror Android 14+), v4l2loopback needs DKMS and boot parameters and is Linux-only, the loopback starves on a static display so first paint took 4–60 s, adb shell input costs ~130 ms per event with no multitouch, long-press, drag or pinch, and the window grab kept a hidden scrcpy window alive. The scrcpy server is the part that is hard to replace and does not need replacing (it creates the virtual display, captures without a consent dialog, encodes on the phone's hardware encoder and injects input); the desktop client is what OCTAVE replaced with code of its own. Result: first paint 0.4–0.8 s, touch at socket latency, and the only external pieces are two Apache-2.0 files OCTAVE ships itself.

Why fork the server

Pinning an upstream jar was enough for the protocol, but the in-car problems (a power press dozing the virtual display, USB dropouts destroying the display and its apps) needed behaviour the phone-side program does not have. Forking under org.octave.phoneserver (commit 200c8c8) gave a place for the MirrorKeeper, the persistent session loop and the TYPE_OCTAVE_* messages (numbered from 100 to stay clear of upstream), while the version string 3.3.4-octave keeps the mismatch check loud. Upstream changes are pulled deliberately and the byte layouts re-verified.

Why a virtual display

new_display=<view size> gives a landscape surface sized for the dash instead of a portrait phone screen, keeps the phone's own screen free (and dark) for the passenger, and on Samsung draws DeX with a taskbar that replaces the nav row. Coordinates are relative to the mirrored display so touch needs no display-id routing. The costs are documented rather than hidden: HOME/APP_SWITCH do nothing there (hence am start --display), the display shares the phone's power group, and Android 11+ is required (the manager falls back to the phone screen below SDK 30). With it the client sends display_ime_policy=local, so the on-screen keyboard opens on the mirrored display rather than on the phone's own screen, where the driver can't see it.

Why the keeper logic lives on the phone

From the dash the reaction is an adb round trip on a 1 s poll: ~1 s of black frames and a visible relight per press, plus adb processes spawned while a session is up. Inside the device a DisplayListener and a 250 ms poll react within a frame or two, the keyguard is read directly from IWindowManager, and the state survives a client reconnect because the keeper is process-wide. OCTAVE keeps only policy (OCTAVE_SET_KEEPER), an escape hatch (OCTAVE_TAKE_BACK) and display (PHONE_STATE).

Why the phone's volume is never touched

Gain is applied to the PCM inside OCTAVE. The alternative -- raising the phone's media volume for the session -- failed silently on the test handset and would leave the phone at maximum after a crash. A crash in the car must not change the phone's state.

Why hold frames instead of clearing the sink

Every transition (doze, wake, reconnect, reattach) was measured with a 30 fps capture of the dash and each sub-second artifact fixed at its source: the last good frame is the correct thing to show until real content arrives, so the client never clears the VideoOutput, black and doze-fade frames are held, and overlays appear only after a delay (1.5 s asleep, 2 s reconnect badge).

Why raw PCM audio

audio_codec=raw means PCM s16le 48 kHz stereo on the wire and no audio decoder in OCTAVE (the code comment on the launch option); the 192 KB/s it costs over USB is small next to the 8 Mbit/s video stream.

Android Auto

Android Auto has two backends behind one manager (androidAutoManager) and one page (frontend/AndroidAutoView.qml), picked by the androidAutoBackend setting (Settings → Accessories → Phone Dock → Android Auto → Head unit):

• Built-in ("native", beta): OCTAVE is the head unit and speaks the Android Auto Protocol to the phone itself. Desktop builds with OpenSSL and FFmpeg only (nativeAvailable); the Head unit choice is hidden where it is missing.
• Google DHU ("dhu", the default): drives Google's Desktop Head Unit, the SDK tool that implements a complete Android Auto head unit against a phone over adb. OCTAVE launches it, sets up its adb forward and the phone-side head unit server, and is meant to show its window inside the dash. A build without the built-in receiver always reports backend as "dhu".

Files:

Logging: categories octave.androidauto (the manager), octave.androidauto.session and octave.androidauto.connection (the built-in receiver), and octave.embeddeddhu. The bottom-bar button is shown when setting androidAutoEnabled is on (Settings → Accessories → Phone Dock → Android Auto → Show in nav bar).

Built-in Android Auto (native receiver)

With androidAutoBackend set to "native", OCTAVE takes the place of the car's head unit. The phone runs Android Auto as it would in a car; the Android Auto app on the phone draws the car screen (Maps, media apps, messaging) and sends it to OCTAVE as H.264 video with PCM audio, and OCTAVE sends touches, night mode and driving status back. Nothing extra runs on the phone and no Google software is needed on the OCTAVE machine.

The link is the phone's Android Auto head unit server, a developer-mode feature of the Android Auto app that listens on phone TCP port 5277 (the socket Google's DHU uses). OCTAVE reaches it through an adb port forward, so it works over a USB cable with USB debugging or over wireless debugging, the same adb connection Phone Mirror uses. Plain "plug in a stock phone" Android Auto over a USB accessory link is not implemented yet.

Checked on a Samsung SM-S948U with Android Auto 17.9 over wireless adb: Maps and Spotify render, taps and drags work, media audio plays.

Phone and OCTAVE setup

On the phone, once

1. adb -- USB debugging on (or the phone paired for wireless debugging in Phone Mirror), exactly as for Phone Mirror.
2. Android Auto developer mode -- open Android Auto's settings on the phone and tap Version about 10 times until it confirms developer mode.
3. Head unit server -- in Android Auto's settings, open the ⋮ menu and tap Start head unit server. It keeps running across USB unplugs and serves one session after another.

In OCTAVE

1. Settings → Accessories → Phone Dock → Android Auto: turn on Show in nav bar (androidAutoEnabled).
2. Under Head unit, pick Built-in (androidAutoBackend = "native"). Only shown when the build has the built-in receiver.
3. Optionally pick a Video size: 800x480, 1280x720 (default) or 1920x1080 (androidAutoResolution). It applies to the next connection. Frame rate (androidAutoFps) and density (androidAutoDpi) have no control yet; edit the settings file (Settings Reference).
4. Provide the head-unit certificate (next section). The Status row on the same settings card shows the certificate folder.
5. Tap Android Auto in the nav bar. The page connects on its own as soon as the phone, its head unit server and the certificate are all there.

The Status row on the settings card shows the manager's nativeStatus (green while running, red for error / needsCertificate), or "Not connected. Open Android Auto from the nav bar to connect." while idle.

The head-unit certificate

Android Auto only accepts a head unit whose TLS client certificate chains to Google's head-unit certificate authority. OCTAVE does not ship such a certificate and never will: nothing in the repository or in the release downloads contains one. The user supplies a PEM certificate and its private key, and is responsible for having the right to use them. OCTAVE looks for them, in order:

1. androidAutoCertPath set: that file. If it contains a PRIVATE KEY block, the key is taken from the same file; otherwise from a sibling file with the same base name and a .key extension (/path/unit.pem → /path/unit.key).
2. androidAutoCertPath empty (the default): headunit.crt and headunit.key in <config dir>/android-auto/ (Linux ~/.config/OCTAVE/android-auto/, Windows %APPDATA%\OCTAVE\android-auto\, macOS ~/Library/Application Support/OCTAVE/android-auto/). The manager's certificateDir property holds this path and both the page and the settings card show it.

The files are read at every connection attempt, so dropping them in while the page shows "Android Auto needs a certificate" is enough: the next retry (within 3 s) picks them up. A certificate OpenSSL can't parse, or a key that doesn't match it, ends the attempt with an error; a certificate the phone does not accept fails the TLS handshake (see Troubleshooting).

What works, what doesn't yet

Which source plays the music

OCTAVE's own player, the mirrored phone and Android Auto can all play music, and exactly one of them owns it at a time: nowPlaying.source ("local", "phone" or "androidauto"). Only a user action changes the owner. A phone connecting, Android Auto starting, a reconnect, or Android Auto's Start music automatically never does. Android Auto asks the head unit before it plays (an audio focus request), so for Android Auto OCTAVE decides; phone mirroring can only be noticed once the phone plays.

Owner and play state are saved as mediaLastSource / mediaLastPlaying (Settings reference), not overwritten by what stops at shutdown or when the phone disconnects. Refusing is done the way a car radio does it: OCTAVE grants the request and at once takes focus back (unsolicited LOSS); answering LOSS makes Android Auto 17.9 ask again about 50 times a second. Phone mirroring keeps its own rule for now (the phone's first media report takes the controls); bringing it under the same rules is in TODO/media-source-arbiter.md. Code: src/managers/nowplaying.cpp (the rules), src/managers/androidautomedia.cpp (Android Auto's side); tests: tst_nowplaying.

What the page shows and what to do

Opening the page calls startNative(). From then on the manager keeps trying until the page's close button (X, top right) calls stopNative(): a missing phone, head unit server or certificate is retried every 3 s, an error after 8 s, while the page is on screen. Leaving the page through the nav bar keeps a running session going (audio continues, video pauses); with no session, the retries pause until the page comes back. Switching Head unit to Google DHU stops a built-in session. A certificate or key OpenSSL can't use is caught before anything connects to the phone. The state is nativeState; the sentence under the title is nativeStatus.

How the built-in receiver works

1. Find the phone (off the GUI thread, QtConcurrent) -- adb devices, then the same serial choice as Phone Mirror (PhoneMirrorManager::pickSerial(), preferring the last phone).
2. Check the head unit server -- adb shell cat /proc/net/tcp6 /proc/net/tcp must show a listening socket on local port 5277 (:149D, state 0A). dumpsys is not used: it can't tell a running service from a stale record.
3. Forward -- adb forward tcp:0 tcp:5277 picks a free local port, so it never collides with a DHU forward or a mirroring tunnel; the forward is removed when the session ends.
4. Session -- an AapConnection on its own thread connects to 127.0.0.1:<port> and runs an AapSession: version exchange, TLS 1.2 with the head-unit certificate (OCTAVE is the TLS client), auth complete, then service discovery, where OCTAVE announces video, three audio sinks, a microphone, a touchscreen with media keys, sensors (driving status, night mode) and media playback status. From TCP connect to discovery it allows 10 s. Every video frame and audio packet is acknowledged; OCTAVE pings every 1.5 s.
5. Media -- H.264 goes through H264Decoder into the page's QVideoSink; PCM goes to one QAudioSink per stream with at most 500 ms queued.
6. Music -- the phone's media focus requests go to AndroidAutoMedia and nowPlaying answers them (above); the media playback status channel brings the track, art and position; OCTAVE's controls go out as key events (KEYCODE_MEDIA_PLAY_PAUSE, NEXT, PREVIOUS).
7. End -- the close button and application quit send a bye-bye and wait for the phone's answer (quit waits up to 3 s): a session cut off mid-stream can leave the phone's head unit server accepting connections but never answering.

Byte layouts and the message flow: docs/ANDROID_AUTO_PROTOCOL.md. Plan, phases and risks: TODO/android-auto-native-receiver.md. The protocol core is tested without a phone by tst_androidauto (Development).

AndroidAutoManager

The built-in backend's members are marked experimental in the published API (Q_CLASSINFO("octave.experimental", ...)): they may still change without a protocol version bump. The mobile stub keeps their names with inert values (backend "dhu", nativeAvailable false, nativeState "idle").

Signals

Properties

Slots and invokables

Not invokable from QML, wired by AppCore: setSettingsManager(), setVolume() / setMuted() (from VolumeController), setAudioOutputDevice() (the audioOutputDevice setting) and setNightMode() (from themeCatalog).

Google DHU backend: what actually works

Google DHU Integration

The DHU is discovered at the standard SDK location (a cached lookup):

# SDK_PATH / extras / google / auto / desktop-head-unit[.exe]

# SDK roots checked:
Windows: %LOCALAPPDATA%\Android\Sdk, %USERPROFILE%\AppData\Local\Android\Sdk, C:\Android\Sdk, C:\Users\%USERNAME%\Android\Sdk
macOS/Linux: ~/Android/Sdk, ~/Library/Android/sdk, /opt/android-sdk, /usr/local/android-sdk

adb comes from the same discovery as the phone mirror: findAdbPath() calls the public static PhoneMirrorManager::findAdb() (cached after the first hit) -- bundled platform-tools, then PATH, then the common SDK locations -- so both features drive the same binary.

DHU Configuration

Before launching, a config file is written to ~/.android/headunit.ini:

[general]
resolution = 1920 1080
dpi = 160

ADB Connection Setup

prepareAdbConnection() runs synchronously on the GUI thread:

1. Check device -- adb devices must list at least one line containing the substring device (no state parsing like the phone mirror's getDeviceState()).
2. Port forwarding -- adb forward --remove tcp:5277 (a stale DHU forward only), then adb forward tcp:5277 tcp:5277 (the DHU's port). An earlier --remove-all also tore down a live phone-mirror localabstract:scrcpy_* tunnel; Android Auto now touches only its own forward, here and in cleanup().
3. Head unit server -- dumpsys activity services com.google.android.projection.gearhead/.companion.DeveloperHeadUnitNetworkService must show a ServiceRecord. If not, OCTAVE tries am start -n com.google.android.projection.gearhead/.companion.MainActivity -a com.google.android.gms.car.action.START_HEAD_UNIT_SERVER (wait 2 s), then am startservice -n ...DeveloperHeadUnitNetworkService (wait 1 s). If neither works it polls once a second for up to 60 s for the user to start "Head unit server" in Android Auto's developer settings, reporting the remaining time every 5 s through connectionProgress.

DhuCapture & Frame Provider

DhuCapture captures a window at 30 FPS on a QTimer into DhuFrameProvider with QScreen::grabWindow(WId) on the primary screen. In practice it never runs against a real DHU window (see What actually works): on Windows capture is never started, and elsewhere startCapture() refuses the handle 0 it is given.

DhuFrameProvider

A QQuickImageProvider registered as image://dhuframe/ in src/main.cpp. It returns the latest frame under a lock (the timer runs on the GUI thread, QML may request images from a render thread), or a black 800x480 placeholder when none is available. No QML file currently references image://dhuframe/.

Launch on Windows

The DHU process is started with CREATE_NO_WINDOW and SW_MINIMIZE (through setCreateProcessArgumentsModifier). There is no window enumeration, so the 100 ms window poll always runs out after 5 s and the process id stands in for the handle.

Touch Input Forwarding

Touch forwarding is not implemented. DhuCapture.sendMouseClick(x, y), reached through the manager's sendDhuClick(), only logs the coordinates and returns, and no QML calls it. The source comment notes that Windows would need SendMessage / mouse_event and that Linux and macOS have no universal API, so ADB-based input would be the route there.

EmbeddedDhuItem

src/items/embeddeddhuitem.{h,cpp} does no window embedding. It connects to the manager's dhuWindowReady / dhuEmbeddedChanged and to DhuCapture.frameReady, exposes isStreaming and a frameCounter property (frameRefresh signal) intended for an Image { source: "image://dhuframe/" + frameCounter } that AndroidAutoView.qml does not contain. detach() / reattach(), called when the view is hidden or shown, only toggle isStreaming.

It connects via connectToManager(androidAutoManager) from the item's Component.onCompleted and picks up a DHU that is already running (with a non-zero handle) when the user navigates back to the view.

iPhone (AirPlay)

An iPhone can't be mirrored the way an Android phone is: iOS has nothing like adb, no program can capture its screen or fake touches from outside, and CarPlay needs Apple's authentication chip. What iOS does have is AirPlay. OCTAVE is an AirPlay receiver (airPlayManager, src/managers/airplaymanager.{h,cpp}), so the iPhone mirrors its screen to OCTAVE, or plays its music through it, the way it would to an Apple TV. AirPlay sends no touch back, so OCTAVE is also a Bluetooth LE mouse: with AssistiveTouch on, a tap on the mirrored screen moves the iPhone's pointer there and clicks. No app on the iPhone, no developer mode, no extra hardware.

Built when libplist, OpenSSL and libavcodec are found (OCTAVE_HAVE_AIRPLAY, Linux desktop and the Pi for now); touch also needs BlueZ (OCTAVE_BLUEZ). airPlayManager.available / touchAvailable say which a build has.

Setting it up

1. Settings > Accessories > Phone Dock > iPhone (AirPlay): turn on AirPlay receiver (airPlayEnabled, off by default: it advertises OCTAVE on the local network). The iPhone page (frontend/AirPlayView.qml, also reached from Phone Mirror's "Using an iPhone?" button) has the same switch.
2. Put the iPhone on the same network as OCTAVE (the page lists OCTAVE's addresses). In the car that means the head unit joins the phone's hotspot, or the other way round.
3. On the iPhone: Control Center > Screen Mirroring > OCTAVE (the airPlayName setting). The first time, the iPhone asks for a PIN, which OCTAVE shows in large digits; it is remembered after that (airPlayRequirePin, on by default). For music only, pick OCTAVE as the speaker in Control Center's music card.
4. When mirroring starts, OCTAVE opens the iPhone page by itself (mirroringStarted). Its X leaves the page; the iPhone keeps mirroring until it is stopped there. The phone button in the nav bar goes back to it.

The receiver's identity (a random, locally administered device id) and the iPhones that paired with the PIN are kept in <config>/airplay/state.json, its pairing key in receiver.pem. Forget paired iPhones clears the list.

Touch through a Bluetooth mouse

With airPlayTouchEnabled on (the default), OCTAVE registers a HID-over-GATT mouse with BlueZ (src/platform/bluezhidmouse.{h,cpp}, D-Bus only, no root) named "OCTAVE Touch". Once:

1. Tap Pair touch (on the iPhone page, the mirrored screen's badge, or the settings card). For three minutes (pairTouch()) OCTAVE accepts a Just Works pairing; at any other time pairing requests are refused, so a stray device (a Bluetooth keyboard) can't pair with the head unit.
2. On the iPhone: Settings > Bluetooth, tap OCTAVE Touch.
3. Settings > Accessibility > Touch > AssistiveTouch: on. The grey pointer appears.

iOS only takes a relative pointer, so src/airplay/pointerpath.{h,cpp} turns a tap into: push the pointer into the top-left corner (40 full-speed reports, after which its position is known), move to the spot in steps of at most 12 counts (small steps keep iOS's pointer acceleration out of the way), press; a drag moves with the button held, and the lift releases. Reports go out every 8 ms. While the position stays known, later taps move from where the last one ended; it is forgotten when the mirrored picture changes size (a rotation) or the mouse reconnects. A mouse wheel on OCTAVE scrolls.

How far the pointer moves per count depends on the iPhone and its AssistiveTouch tracking speed, so it is calibrated: airPlayTouchScale is the number of counts across the screen's short side (600 to start). Calibrate touch on the mirrored screen homes the pointer and moves it 240 counts right and down; tap the pointer where it shows and the scale is worked out from both axes (AirPlayManager::scaleFromCalibration).

What the page shows

How it works

The protocol is UxPlay's library, compiled in from third_party/uxplay/ as octave_uxplay (RTSP over llhttp, legacy and AirPlay 2 pairing, FairPlay setup, the mirror and audio RTP streams, NTP timing, and its own mDNS responder, so no Avahi or Bonjour is needed). It advertises _airplay._tcp and _raop._tcp with UxPlay's mirroring + audio feature set, without AirPlay video (HLS) or H.265. src/airplay/airplayreceiver.{h,cpp} does what UxPlay's GStreamer renderers did:

• Video: the Annex B H.264 access units go through the shared H264Decoder on UxPlay's mirror thread; the newest frame goes to the page's VideoOutput sink, as with phone mirroring and Android Auto. The display size offered to the iPhone is airPlayResolution (1280x720 at 30 fps by default).
• Audio: ALAC (music), AAC-LC or AAC-ELD (mirroring) through libavcodec into a QAudioSink on the picked output, at OCTAVE's volume (VolumeController) times the iPhone's own slider through the same curve. Muting OCTAVE mutes it.
• Track info (DMAP minm / asar / asal) and cover art go to title / artist / album and AirPlayManager::coverArt().

Checked without an iPhone: tst_airplay starts a real receiver and reads its RTSP /info, and a running OCTAVE answers mDNS queries for both services.

What is not checked yet

Everything that needs an iPhone (TODO/iphone-airplay.md): the PIN pairing, the picture and audio, the Bluetooth mouse pairing and reconnecting after a restart, the pointer path and the calibration, and the head unit's software decode at 1280x720. AirPlay audio does not take part in nowPlaying's one-source rules yet, so local music keeps playing under it. CarPlay is a separate plan (TODO/carplay-native.md).

Custom QML Types

One custom type remains. It is registered in src/main.cpp:

// src/main.cpp
qmlRegisterType<EmbeddedDhuItem>("OCTAVE.AndroidAuto", 1, 0, "EmbeddedDhuItem");

QML imports it as import OCTAVE.AndroidAuto 1.0. The former OCTAVE.PhoneMirror module (EmbeddedScrcpyItem, ScrcpyHostItem, ScrcpyCaptureItem) was removed with the scrcpy-binary path; Phone Mirror uses a plain QtMultimedia VideoOutput. On Android the build registers a stub EmbeddedDhuItem so AndroidAutoView.qml still parses.

Image Providers

Phone Mirror does not use an image provider: decoded frames are pushed as QVideoFrames into the QVideoSink of the view's VideoOutput, which avoids a per-frame RGB conversion and QImage copy. The intended usage pattern for the remaining provider, should a view adopt it:

// Refresh the image by changing the source URL
Image {
    source: "image://dhuframe/" + embeddedDhu.frameCounter
    cache: false
}