Files
Adam Shiervani 90750bf609 Experimental Audio Support (#1475)
* WIP: USB audio capture via UAC1 gadget with G.722 + PCMU encoding

Adds ALSA capture from the USB UAC1 gadget, G.722/PCMU encoding, a WebRTC
audio track, and an e2e remote-agent flow that plays a tone on the remote
host and verifies it reaches the browser.

Snapshot of codex-driven implementation before simplification.

* Simplify USB audio path: drop residual scaffolding from long session

Backend:
- audio.go: drop source rotation + "no-data reopen with next device" loop.
  One source (UAC1Gadget; falls back to hw:1,0 only if sysfs lookup fails).
- internal/audio: remove unused Reader interface and unavailableCapture stub.
  Stub now returns the concrete type with an error.
- webrtc.go: inline single-use resolveAudioCodec helper; DRY video/audio RTCP
  drain into drainRTCP; fold startSessionAudio into the connect callback.

Frontend:
- devices.$id.tsx: drop remoteMediaStreamRef track-merging. Backend tracks
  share stream ID "kvm", so pion delivers them in one MediaStream — just
  assign event.streams[0].
- WebRTCVideo.tsx: replace dynamic per-track <audio> creation + ref array
  with a single hidden <audio> bound to mediaStream.

Remote agent:
- Drop PipeWire/wpctl detection path; plughw: works directly.
- Drop killStaleAudioToneProcesses pkill workaround; the (cmd, cancel, done)
  trio collapses to a single *exec.Cmd field with Start/Kill/Wait.

E2E:
- ra-audio.spec.ts: drop attachAudioDiagnostics scaffold and openReadyPage
  duplicate. Spec is now linear: setup → wait for track → diff stats → tone.

Net: ~355 LOC removed.

* Fix onSessionConnected race; replace onFirst/CurrentSessionConnected split

The previous codex split routed audio start through onCurrentSessionConnected,
gated on session == currentSession. But currentSession is assigned by the
caller (web.go, cloud.go) AFTER ExchangeOffer returns, while
OnICEConnectionStateChange can fire from inside ExchangeOffer or shortly
after — racing the assignment. When the race hits, the equality check fails,
the callback is skipped, and audio never starts.

Pass session into the callback directly so the per-session setup uses the
session in hand, not whatever currentSession happens to point to at that
instant. Keep stopVideoSleepModeTicker on the count-edge (still only on
first-session) and let onSessionConnected handle the rest unconditionally.

* Pin WebRTC playout-delay to zero so receivers don't ratchet under stress

Chrome's adaptive receive-side jitter buffer grows under stress (e.g.
playing a video on the controlled machine) and does not reliably shrink
back; the Connection Stats "Playback Delay" graph used to climb to
~300 ms and stay there until the page was reloaded.

The trigger is the USB UAC1 audio path, not video motion per se — once
real audio starts flowing, Chrome's AV-sync layer pulls the video
jitter buffer up to whatever the audio path settles at, and the ratchet
locks in. Receiver-side hints (jitterBufferTarget, playoutDelayHint,
setMinimumJitterBufferDelay) cap the steady state but don't release a
buffer that has already grown.

Fix: register the WebRTC playout-delay RTP header extension on both
audio and video and stamp min=max=0 on every outgoing packet via a
pion interceptor. Chrome treats this as an authoritative override of
its adaptive logic and keeps both buffers at the decoder floor through
and after stress, with no peer-connection rebuild needed.

Test: drive the host display with a real audio+video file via
gst-launch playbin (audio routed through PipeWire to the USB UAC1
sink) and assert receive-side video delay stays bounded both during
and after playback.

* Simplify audio capture loop and ALSA reader

- Drop short-read zero-fill in ALSA reader; return ErrNoAudioData so the
  capture loop emits no frame for the cycle instead of half-silent audio.
- Replace ErrNoAudioData = io.ErrNoProgress (wrong semantic) with a domain
  sentinel and remove the unused idleReads debug counter.
- Encoders sum all source samples before one divide — better precision,
  fewer ops; clampS16 and sampleS16 helpers gone.
- Resolve audio codec inline in runAudioCapture; drop the audioCodecForTrack
  wrapper. Caller checks AudioTrack != nil so startAudio no longer accepts
  nil as a stop signal.
- Use C.GoString instead of hand-rolled cString helper.
- Add a why-comment on the separate <audio> element (video stays muted).

* Trim playoutdelay per-symbol comments

Keep the package-level "why" (Chrome's one-way jitter buffer); drop
restate-the-signature comments on Factory, NewFactory, NewInterceptor,
and BindLocalStream.

* e2e: extract shared audio helpers and drop dev-only override

- Lift ensureNoPasswordViaAPI and waitForAudioStream into helpers.ts (the
  audio spec was inlining both, the latter as a copy of waitForVideoStream).
- ra-audio.spec.ts shrinks from 78 to 55 lines.
- Remove the JETKVM_AUDIO_DEVICE override in the remote agent: it fabricated
  an AudioDeviceInfo with is_jetkvm=true regardless of what device the env
  var pointed at, silently lying to the spec's assertion. Audio device
  discovery via aplay + /proc/asound/.../usbid is reliable; if no JetKVM
  device is present the spec already skips.

* Reopen ALSA capture on persistent read errors

The C-side recovers EPIPE/ESTRPIPE via snd_pcm_recover; the errors that
surface to Go (EBADFD, ENODEV, …) usually mean the handle is dead —
typically a USB gadget rebuild or host reattach mid-session, which used
to leave audio silent until the session disconnected.

After 5 consecutive non-idle read errors, close and reopen the capture
with exponential backoff (100 ms → 2 s cap). Initial open uses the same
helper so we keep retrying instead of giving up if the gadget isn't ready
yet. Re-resolves the card each attempt so a USB re-enumeration that
shifts the card number is picked up automatically.

* Add Audio settings page with Enable Audio toggle (experimental)

Audio is opt-in via device config. New Audio nav entry in Settings sits
next to Video, with a single "Enable Audio" item marked Experimental
(mirrors HTTPS Mode in Access).

Backend:
- Config.AudioEnabled (default false), persisted to /userdata/kvm_config.json
- getAudioConfig / setAudioConfig JSON-RPC handlers
- webrtc.go: extract attachAudioTrack helper; skip track creation when
  disabled or when the offer advertises no supported codec. The SDP
  answer leaves the audio m-line inactive, so flipping the toggle
  requires a fresh connection (page reload).

Frontend:
- New devices.$id.settings.audio.tsx — fetches state via getAudioConfig,
  saves via setAudioConfig, optimistic UI with rollback on error.
- devices.$id.tsx always offers audio in the SDP; backend decides.
- en.json + 13 locale files: 5 keys each (audio_*, settings_audio) with
  proper translations honoring per-language formality.

E2E:
- ra-audio.spec.ts: connect, enable via RPC, reload, verify audio energy.
  Restores disabled state in a finally block so other specs aren't
  affected. 9 s on kvm-2 + .180.

* Reload page after enabling audio so the browser prompts for autoplay

Re-negotiation only happens on a fresh WebRTC session, and the autoplay
overlay needs a user gesture to play the new audio track. A simple reload
covers both — the toggle's user click acts as the gesture, the new offer
includes audio, and the overlay surfaces normally.

Disable path is unchanged (audio stops naturally on the next connect).

* Reload page on disable too so audio stops immediately

* Trim audio_* copy across all 14 locales

Page header and item description previously said roughly the same thing
in long form. Now: page-level describes the topic ("Stream audio from
the host to your browser"); item-level is a terse one-liner ("Stream
HDMI audio from the host."). Drops the "Requires a fresh connection"
clause — the page auto-reloads on toggle, so it's no longer accurate.

Per-language tone follows I18N_BEST_PRACTICES.md: formal Sie/vous/usted/
вы/chi (de/fr/es/ru/cy), informal du (sv/nb/da), polite です/ます (ja),
infinitive (it), European Portuguese (pt).

* Fold audio e2e into the standard remote-agent project

Drop the dedicated test_audio_e2e Makefile target and the separate
remote-agent-audio Playwright project. The remote-agent project now
matches every ra-*.spec.ts under e2e/remote-agent, so make test_e2e
runs ra-audio.spec.ts alongside ra-all.spec.ts in the same worker.

* Fall back to bare-track MediaStream when ontrack streams[] is empty

Hit a state where pc.getReceivers() showed live video and audio tracks
but useRTCStore.mediaStream stayed undefined — the SDP answer arrived
without a=msid, so event.streams[0] was undefined and
setMediaStream(undefined) left the store empty even though RTP was
flowing. Only a hard reload recovered.

Now: when the event carries a stream, use it as before. When it doesn't,
get-or-create a MediaStream and append the track. Re-using the existing
store value across both ontrack invocations keeps audio + video on the
same MediaStream so the autoplay/video pipeline downstream is unchanged.

* Close peer connection before reload-on-toggle

Firefox's soft reload doesn't always tear down the RTCPeerConnection,
which leaves the post-reload page in a half-renegotiated state: tracks
arrive on receivers but never attach to a MediaStream, so video stays
stuck on "Loading…" (or the page falls back to the pre-connect blue
background) until a hard refresh. Closing the PC explicitly before
reload guarantees a clean start.

* Aggregate ontrack into one canonical MediaStream

The answer SDP from pion omits a=msid for the audio track in some
configurations (visible on Firefox: video keeps its msid, audio doesn't).
The previous handler called setMediaStream(event.streams[0]) on each
ontrack, so:

  1. video ontrack → setMediaStream(streamA)  [has video]
  2. audio ontrack → setMediaStream(streamB)  [synthetic, audio only]

streamB replaces streamA, video disappears. Hard refresh only "fixed" it
incidentally — the same SDP would break the next negotiation too.

Now: ignore event.streams[0], maintain one canonical MediaStream in the
store, and addTrack into it on every ontrack. Browsers render tracks
added to a live MediaStream that's already attached to srcObject, so
both audio and video stay attached regardless of which order ontrack
fires or whether the SDP carried msid.

* Only render <audio> element when device audio is enabled

The backend keeps the m=audio section in the SDP even when audio is
disabled (just inactive direction), so Firefox still attaches a muted
audio track to the MediaStream. The autoplay <audio> element then
triggers Firefox's "block audio" policy on a stream that will never
actually play any sound.

Fetch getAudioConfig once the RPC channel is up, then conditionally
render the <audio> element. No autoplay prompt when audio is off.

* Drop stale mediaStream on reconnect; release audio capture when owner session ends

* Back off briefly on idle ALSA reads to avoid CPU spin

* Re-wire <audio> when track arrives late; gate VideoStart to first session
2026-05-22 10:43:30 +02:00

195 lines
4.7 KiB
Go

package kvm
import (
"context"
"errors"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/jetkvm/kvm/internal/audio"
"github.com/pion/webrtc/v4"
"github.com/pion/webrtc/v4/pkg/media"
)
var (
audioCancel context.CancelFunc
audioStopped chan struct{}
audioTrack *webrtc.TrackLocalStaticSample
audioMu sync.Mutex
)
func startAudio(track *webrtc.TrackLocalStaticSample) {
audioMu.Lock()
defer audioMu.Unlock()
stopAudioLocked()
ctx, cancel := context.WithCancel(context.Background())
audioCancel = cancel
audioStopped = make(chan struct{})
audioTrack = track
go runAudioCapture(ctx, track, audioStopped)
}
func stopAudio() {
audioMu.Lock()
defer audioMu.Unlock()
stopAudioLocked()
}
// stopAudioIfOwner stops the audio capture only if it is currently bound to
// track. Used on session teardown so capture doesn't keep writing samples to
// a track whose peer connection has closed.
func stopAudioIfOwner(track *webrtc.TrackLocalStaticSample) {
audioMu.Lock()
defer audioMu.Unlock()
if audioTrack != track {
return
}
stopAudioLocked()
}
func stopAudioLocked() {
if audioCancel == nil {
return
}
audioCancel()
<-audioStopped
audioCancel = nil
audioStopped = nil
audioTrack = nil
}
// reopenThreshold is the number of consecutive non-idle read errors that
// triggers a close+reopen of the ALSA handle. The C-side already recovers
// EPIPE/ESTRPIPE; errors that surface here (EBADFD, ENODEV, …) usually mean
// the handle is dead — typically a USB gadget rebuild or host reattach.
const reopenThreshold = 5
func runAudioCapture(ctx context.Context, track *webrtc.TrackLocalStaticSample, stopped chan<- struct{}) {
defer close(stopped)
codec := audio.CodecPCMU
if strings.EqualFold(track.Codec().MimeType, webrtc.MimeTypeG722) {
codec = audio.CodecG722
}
capture, err := openCaptureWithBackoff(ctx)
if err != nil {
return
}
defer func() { capture.Close() }()
audioLogger.Info().Str("codec", codec.String()).Msg("audio capture started")
defer audioLogger.Info().Msg("audio capture stopped")
sample := media.Sample{Duration: 20 * time.Millisecond}
consecutiveErrors := 0
for {
select {
case <-ctx.Done():
return
default:
}
payload, err := capture.ReadEncoded(codec)
if err != nil {
if errors.Is(err, audio.ErrNoAudioData) {
// Partial period or idle ALSA — back off ~half a frame so we
// don't spin while the buffer fills.
select {
case <-ctx.Done():
return
case <-time.After(10 * time.Millisecond):
}
continue
}
consecutiveErrors++
audioLogger.Warn().Err(err).Int("errs", consecutiveErrors).Msg("audio capture read failed")
if consecutiveErrors >= reopenThreshold {
capture.Close()
next, err := openCaptureWithBackoff(ctx)
if err != nil {
return
}
capture = next
consecutiveErrors = 0
continue
}
time.Sleep(100 * time.Millisecond)
continue
}
consecutiveErrors = 0
if len(payload) == 0 {
continue
}
sample.Data = payload
if err := track.WriteSample(sample); err != nil {
audioLogger.Warn().Err(err).Msg("audio sample write failed")
time.Sleep(100 * time.Millisecond)
}
}
}
// openCaptureWithBackoff opens the ALSA capture device, retrying with
// exponential backoff (capped at 2 s) until success or ctx cancellation.
// Re-resolves the card on every attempt so a USB re-enumeration that hands
// the gadget a new card number is picked up automatically.
func openCaptureWithBackoff(ctx context.Context) (*audio.ALSACapture, error) {
backoff := 100 * time.Millisecond
for {
device := alsaCaptureDevice()
capture, err := audio.OpenALSACapture(device)
if err == nil {
audioLogger.Info().Str("device", device).Msg("audio capture opened")
return capture, nil
}
audioLogger.Warn().Err(err).Str("device", device).Dur("retry_in", backoff).Msg("audio capture open failed")
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(backoff):
}
if backoff *= 2; backoff > 2*time.Second {
backoff = 2 * time.Second
}
}
}
// alsaCaptureDevice returns the ALSA device for the UAC1 gadget card.
func alsaCaptureDevice() string {
if card, ok := findALSACard("UAC1Gadget"); ok {
return "hw:" + strconv.Itoa(card) + ",0"
}
return "hw:1,0"
}
func findALSACard(cardID string) (int, bool) {
entries, err := os.ReadDir("/sys/class/sound")
if err != nil {
return 0, false
}
for _, entry := range entries {
name := entry.Name()
if !strings.HasPrefix(name, "card") {
continue
}
id, err := os.ReadFile(filepath.Join("/sys/class/sound", name, "id"))
if err != nil || strings.TrimSpace(string(id)) != cardID {
continue
}
if card, err := strconv.Atoi(strings.TrimPrefix(name, "card")); err == nil {
return card, true
}
}
return 0, false
}