mirror of
https://github.com/jetkvm/kvm.git
synced 2026-06-20 05:24:32 +00:00
Adam Shiervani
90750bf609
* WIP: USB audio capture via UAC1 gadget with G.722 + PCMU encoding
Adds ALSA capture from the USB UAC1 gadget, G.722/PCMU encoding, a WebRTC
audio track, and an e2e remote-agent flow that plays a tone on the remote
host and verifies it reaches the browser.
Snapshot of codex-driven implementation before simplification.
* Simplify USB audio path: drop residual scaffolding from long session
Backend:
- audio.go: drop source rotation + "no-data reopen with next device" loop.
One source (UAC1Gadget; falls back to hw:1,0 only if sysfs lookup fails).
- internal/audio: remove unused Reader interface and unavailableCapture stub.
Stub now returns the concrete type with an error.
- webrtc.go: inline single-use resolveAudioCodec helper; DRY video/audio RTCP
drain into drainRTCP; fold startSessionAudio into the connect callback.
Frontend:
- devices.$id.tsx: drop remoteMediaStreamRef track-merging. Backend tracks
share stream ID "kvm", so pion delivers them in one MediaStream — just
assign event.streams[0].
- WebRTCVideo.tsx: replace dynamic per-track <audio> creation + ref array
with a single hidden <audio> bound to mediaStream.
Remote agent:
- Drop PipeWire/wpctl detection path; plughw: works directly.
- Drop killStaleAudioToneProcesses pkill workaround; the (cmd, cancel, done)
trio collapses to a single *exec.Cmd field with Start/Kill/Wait.
E2E:
- ra-audio.spec.ts: drop attachAudioDiagnostics scaffold and openReadyPage
duplicate. Spec is now linear: setup → wait for track → diff stats → tone.
Net: ~355 LOC removed.
* Fix onSessionConnected race; replace onFirst/CurrentSessionConnected split
The previous codex split routed audio start through onCurrentSessionConnected,
gated on session == currentSession. But currentSession is assigned by the
caller (web.go, cloud.go) AFTER ExchangeOffer returns, while
OnICEConnectionStateChange can fire from inside ExchangeOffer or shortly
after — racing the assignment. When the race hits, the equality check fails,
the callback is skipped, and audio never starts.
Pass session into the callback directly so the per-session setup uses the
session in hand, not whatever currentSession happens to point to at that
instant. Keep stopVideoSleepModeTicker on the count-edge (still only on
first-session) and let onSessionConnected handle the rest unconditionally.
* Pin WebRTC playout-delay to zero so receivers don't ratchet under stress
Chrome's adaptive receive-side jitter buffer grows under stress (e.g.
playing a video on the controlled machine) and does not reliably shrink
back; the Connection Stats "Playback Delay" graph used to climb to
~300 ms and stay there until the page was reloaded.
The trigger is the USB UAC1 audio path, not video motion per se — once
real audio starts flowing, Chrome's AV-sync layer pulls the video
jitter buffer up to whatever the audio path settles at, and the ratchet
locks in. Receiver-side hints (jitterBufferTarget, playoutDelayHint,
setMinimumJitterBufferDelay) cap the steady state but don't release a
buffer that has already grown.
Fix: register the WebRTC playout-delay RTP header extension on both
audio and video and stamp min=max=0 on every outgoing packet via a
pion interceptor. Chrome treats this as an authoritative override of
its adaptive logic and keeps both buffers at the decoder floor through
and after stress, with no peer-connection rebuild needed.
Test: drive the host display with a real audio+video file via
gst-launch playbin (audio routed through PipeWire to the USB UAC1
sink) and assert receive-side video delay stays bounded both during
and after playback.
* Simplify audio capture loop and ALSA reader
- Drop short-read zero-fill in ALSA reader; return ErrNoAudioData so the
capture loop emits no frame for the cycle instead of half-silent audio.
- Replace ErrNoAudioData = io.ErrNoProgress (wrong semantic) with a domain
sentinel and remove the unused idleReads debug counter.
- Encoders sum all source samples before one divide — better precision,
fewer ops; clampS16 and sampleS16 helpers gone.
- Resolve audio codec inline in runAudioCapture; drop the audioCodecForTrack
wrapper. Caller checks AudioTrack != nil so startAudio no longer accepts
nil as a stop signal.
- Use C.GoString instead of hand-rolled cString helper.
- Add a why-comment on the separate <audio> element (video stays muted).
* Trim playoutdelay per-symbol comments
Keep the package-level "why" (Chrome's one-way jitter buffer); drop
restate-the-signature comments on Factory, NewFactory, NewInterceptor,
and BindLocalStream.
* e2e: extract shared audio helpers and drop dev-only override
- Lift ensureNoPasswordViaAPI and waitForAudioStream into helpers.ts (the
audio spec was inlining both, the latter as a copy of waitForVideoStream).
- ra-audio.spec.ts shrinks from 78 to 55 lines.
- Remove the JETKVM_AUDIO_DEVICE override in the remote agent: it fabricated
an AudioDeviceInfo with is_jetkvm=true regardless of what device the env
var pointed at, silently lying to the spec's assertion. Audio device
discovery via aplay + /proc/asound/.../usbid is reliable; if no JetKVM
device is present the spec already skips.
* Reopen ALSA capture on persistent read errors
The C-side recovers EPIPE/ESTRPIPE via snd_pcm_recover; the errors that
surface to Go (EBADFD, ENODEV, …) usually mean the handle is dead —
typically a USB gadget rebuild or host reattach mid-session, which used
to leave audio silent until the session disconnected.
After 5 consecutive non-idle read errors, close and reopen the capture
with exponential backoff (100 ms → 2 s cap). Initial open uses the same
helper so we keep retrying instead of giving up if the gadget isn't ready
yet. Re-resolves the card each attempt so a USB re-enumeration that
shifts the card number is picked up automatically.
* Add Audio settings page with Enable Audio toggle (experimental)
Audio is opt-in via device config. New Audio nav entry in Settings sits
next to Video, with a single "Enable Audio" item marked Experimental
(mirrors HTTPS Mode in Access).
Backend:
- Config.AudioEnabled (default false), persisted to /userdata/kvm_config.json
- getAudioConfig / setAudioConfig JSON-RPC handlers
- webrtc.go: extract attachAudioTrack helper; skip track creation when
disabled or when the offer advertises no supported codec. The SDP
answer leaves the audio m-line inactive, so flipping the toggle
requires a fresh connection (page reload).
Frontend:
- New devices.$id.settings.audio.tsx — fetches state via getAudioConfig,
saves via setAudioConfig, optimistic UI with rollback on error.
- devices.$id.tsx always offers audio in the SDP; backend decides.
- en.json + 13 locale files: 5 keys each (audio_*, settings_audio) with
proper translations honoring per-language formality.
E2E:
- ra-audio.spec.ts: connect, enable via RPC, reload, verify audio energy.
Restores disabled state in a finally block so other specs aren't
affected. 9 s on kvm-2 + .180.
* Reload page after enabling audio so the browser prompts for autoplay
Re-negotiation only happens on a fresh WebRTC session, and the autoplay
overlay needs a user gesture to play the new audio track. A simple reload
covers both — the toggle's user click acts as the gesture, the new offer
includes audio, and the overlay surfaces normally.
Disable path is unchanged (audio stops naturally on the next connect).
* Reload page on disable too so audio stops immediately
* Trim audio_* copy across all 14 locales
Page header and item description previously said roughly the same thing
in long form. Now: page-level describes the topic ("Stream audio from
the host to your browser"); item-level is a terse one-liner ("Stream
HDMI audio from the host."). Drops the "Requires a fresh connection"
clause — the page auto-reloads on toggle, so it's no longer accurate.
Per-language tone follows I18N_BEST_PRACTICES.md: formal Sie/vous/usted/
вы/chi (de/fr/es/ru/cy), informal du (sv/nb/da), polite です/ます (ja),
infinitive (it), European Portuguese (pt).
* Fold audio e2e into the standard remote-agent project
Drop the dedicated test_audio_e2e Makefile target and the separate
remote-agent-audio Playwright project. The remote-agent project now
matches every ra-*.spec.ts under e2e/remote-agent, so make test_e2e
runs ra-audio.spec.ts alongside ra-all.spec.ts in the same worker.
* Fall back to bare-track MediaStream when ontrack streams[] is empty
Hit a state where pc.getReceivers() showed live video and audio tracks
but useRTCStore.mediaStream stayed undefined — the SDP answer arrived
without a=msid, so event.streams[0] was undefined and
setMediaStream(undefined) left the store empty even though RTP was
flowing. Only a hard reload recovered.
Now: when the event carries a stream, use it as before. When it doesn't,
get-or-create a MediaStream and append the track. Re-using the existing
store value across both ontrack invocations keeps audio + video on the
same MediaStream so the autoplay/video pipeline downstream is unchanged.
* Close peer connection before reload-on-toggle
Firefox's soft reload doesn't always tear down the RTCPeerConnection,
which leaves the post-reload page in a half-renegotiated state: tracks
arrive on receivers but never attach to a MediaStream, so video stays
stuck on "Loading…" (or the page falls back to the pre-connect blue
background) until a hard refresh. Closing the PC explicitly before
reload guarantees a clean start.
* Aggregate ontrack into one canonical MediaStream
The answer SDP from pion omits a=msid for the audio track in some
configurations (visible on Firefox: video keeps its msid, audio doesn't).
The previous handler called setMediaStream(event.streams[0]) on each
ontrack, so:
1. video ontrack → setMediaStream(streamA) [has video]
2. audio ontrack → setMediaStream(streamB) [synthetic, audio only]
streamB replaces streamA, video disappears. Hard refresh only "fixed" it
incidentally — the same SDP would break the next negotiation too.
Now: ignore event.streams[0], maintain one canonical MediaStream in the
store, and addTrack into it on every ontrack. Browsers render tracks
added to a live MediaStream that's already attached to srcObject, so
both audio and video stay attached regardless of which order ontrack
fires or whether the SDP carried msid.
* Only render <audio> element when device audio is enabled
The backend keeps the m=audio section in the SDP even when audio is
disabled (just inactive direction), so Firefox still attaches a muted
audio track to the MediaStream. The autoplay <audio> element then
triggers Firefox's "block audio" policy on a stream that will never
actually play any sound.
Fetch getAudioConfig once the RPC channel is up, then conditionally
render the <audio> element. No autoplay prompt when audio is off.
* Drop stale mediaStream on reconnect; release audio capture when owner session ends
* Back off briefly on idle ALSA reads to avoid CPU spin
* Re-wire <audio> when track arrives late; gate VideoStart to first session