mirror of
https://github.com/jetkvm/kvm.git
synced 2026-06-20 05:24:32 +00:00
90750bf609
* WIP: USB audio capture via UAC1 gadget with G.722 + PCMU encoding
Adds ALSA capture from the USB UAC1 gadget, G.722/PCMU encoding, a WebRTC
audio track, and an e2e remote-agent flow that plays a tone on the remote
host and verifies it reaches the browser.
Snapshot of codex-driven implementation before simplification.
* Simplify USB audio path: drop residual scaffolding from long session
Backend:
- audio.go: drop source rotation + "no-data reopen with next device" loop.
One source (UAC1Gadget; falls back to hw:1,0 only if sysfs lookup fails).
- internal/audio: remove unused Reader interface and unavailableCapture stub.
Stub now returns the concrete type with an error.
- webrtc.go: inline single-use resolveAudioCodec helper; DRY video/audio RTCP
drain into drainRTCP; fold startSessionAudio into the connect callback.
Frontend:
- devices.$id.tsx: drop remoteMediaStreamRef track-merging. Backend tracks
share stream ID "kvm", so pion delivers them in one MediaStream — just
assign event.streams[0].
- WebRTCVideo.tsx: replace dynamic per-track <audio> creation + ref array
with a single hidden <audio> bound to mediaStream.
Remote agent:
- Drop PipeWire/wpctl detection path; plughw: works directly.
- Drop killStaleAudioToneProcesses pkill workaround; the (cmd, cancel, done)
trio collapses to a single *exec.Cmd field with Start/Kill/Wait.
E2E:
- ra-audio.spec.ts: drop attachAudioDiagnostics scaffold and openReadyPage
duplicate. Spec is now linear: setup → wait for track → diff stats → tone.
Net: ~355 LOC removed.
* Fix onSessionConnected race; replace onFirst/CurrentSessionConnected split
The previous codex split routed audio start through onCurrentSessionConnected,
gated on session == currentSession. But currentSession is assigned by the
caller (web.go, cloud.go) AFTER ExchangeOffer returns, while
OnICEConnectionStateChange can fire from inside ExchangeOffer or shortly
after — racing the assignment. When the race hits, the equality check fails,
the callback is skipped, and audio never starts.
Pass session into the callback directly so the per-session setup uses the
session in hand, not whatever currentSession happens to point to at that
instant. Keep stopVideoSleepModeTicker on the count-edge (still only on
first-session) and let onSessionConnected handle the rest unconditionally.
* Pin WebRTC playout-delay to zero so receivers don't ratchet under stress
Chrome's adaptive receive-side jitter buffer grows under stress (e.g.
playing a video on the controlled machine) and does not reliably shrink
back; the Connection Stats "Playback Delay" graph used to climb to
~300 ms and stay there until the page was reloaded.
The trigger is the USB UAC1 audio path, not video motion per se — once
real audio starts flowing, Chrome's AV-sync layer pulls the video
jitter buffer up to whatever the audio path settles at, and the ratchet
locks in. Receiver-side hints (jitterBufferTarget, playoutDelayHint,
setMinimumJitterBufferDelay) cap the steady state but don't release a
buffer that has already grown.
Fix: register the WebRTC playout-delay RTP header extension on both
audio and video and stamp min=max=0 on every outgoing packet via a
pion interceptor. Chrome treats this as an authoritative override of
its adaptive logic and keeps both buffers at the decoder floor through
and after stress, with no peer-connection rebuild needed.
Test: drive the host display with a real audio+video file via
gst-launch playbin (audio routed through PipeWire to the USB UAC1
sink) and assert receive-side video delay stays bounded both during
and after playback.
* Simplify audio capture loop and ALSA reader
- Drop short-read zero-fill in ALSA reader; return ErrNoAudioData so the
capture loop emits no frame for the cycle instead of half-silent audio.
- Replace ErrNoAudioData = io.ErrNoProgress (wrong semantic) with a domain
sentinel and remove the unused idleReads debug counter.
- Encoders sum all source samples before one divide — better precision,
fewer ops; clampS16 and sampleS16 helpers gone.
- Resolve audio codec inline in runAudioCapture; drop the audioCodecForTrack
wrapper. Caller checks AudioTrack != nil so startAudio no longer accepts
nil as a stop signal.
- Use C.GoString instead of hand-rolled cString helper.
- Add a why-comment on the separate <audio> element (video stays muted).
* Trim playoutdelay per-symbol comments
Keep the package-level "why" (Chrome's one-way jitter buffer); drop
restate-the-signature comments on Factory, NewFactory, NewInterceptor,
and BindLocalStream.
* e2e: extract shared audio helpers and drop dev-only override
- Lift ensureNoPasswordViaAPI and waitForAudioStream into helpers.ts (the
audio spec was inlining both, the latter as a copy of waitForVideoStream).
- ra-audio.spec.ts shrinks from 78 to 55 lines.
- Remove the JETKVM_AUDIO_DEVICE override in the remote agent: it fabricated
an AudioDeviceInfo with is_jetkvm=true regardless of what device the env
var pointed at, silently lying to the spec's assertion. Audio device
discovery via aplay + /proc/asound/.../usbid is reliable; if no JetKVM
device is present the spec already skips.
* Reopen ALSA capture on persistent read errors
The C-side recovers EPIPE/ESTRPIPE via snd_pcm_recover; the errors that
surface to Go (EBADFD, ENODEV, …) usually mean the handle is dead —
typically a USB gadget rebuild or host reattach mid-session, which used
to leave audio silent until the session disconnected.
After 5 consecutive non-idle read errors, close and reopen the capture
with exponential backoff (100 ms → 2 s cap). Initial open uses the same
helper so we keep retrying instead of giving up if the gadget isn't ready
yet. Re-resolves the card each attempt so a USB re-enumeration that
shifts the card number is picked up automatically.
* Add Audio settings page with Enable Audio toggle (experimental)
Audio is opt-in via device config. New Audio nav entry in Settings sits
next to Video, with a single "Enable Audio" item marked Experimental
(mirrors HTTPS Mode in Access).
Backend:
- Config.AudioEnabled (default false), persisted to /userdata/kvm_config.json
- getAudioConfig / setAudioConfig JSON-RPC handlers
- webrtc.go: extract attachAudioTrack helper; skip track creation when
disabled or when the offer advertises no supported codec. The SDP
answer leaves the audio m-line inactive, so flipping the toggle
requires a fresh connection (page reload).
Frontend:
- New devices.$id.settings.audio.tsx — fetches state via getAudioConfig,
saves via setAudioConfig, optimistic UI with rollback on error.
- devices.$id.tsx always offers audio in the SDP; backend decides.
- en.json + 13 locale files: 5 keys each (audio_*, settings_audio) with
proper translations honoring per-language formality.
E2E:
- ra-audio.spec.ts: connect, enable via RPC, reload, verify audio energy.
Restores disabled state in a finally block so other specs aren't
affected. 9 s on kvm-2 + .180.
* Reload page after enabling audio so the browser prompts for autoplay
Re-negotiation only happens on a fresh WebRTC session, and the autoplay
overlay needs a user gesture to play the new audio track. A simple reload
covers both — the toggle's user click acts as the gesture, the new offer
includes audio, and the overlay surfaces normally.
Disable path is unchanged (audio stops naturally on the next connect).
* Reload page on disable too so audio stops immediately
* Trim audio_* copy across all 14 locales
Page header and item description previously said roughly the same thing
in long form. Now: page-level describes the topic ("Stream audio from
the host to your browser"); item-level is a terse one-liner ("Stream
HDMI audio from the host."). Drops the "Requires a fresh connection"
clause — the page auto-reloads on toggle, so it's no longer accurate.
Per-language tone follows I18N_BEST_PRACTICES.md: formal Sie/vous/usted/
вы/chi (de/fr/es/ru/cy), informal du (sv/nb/da), polite です/ます (ja),
infinitive (it), European Portuguese (pt).
* Fold audio e2e into the standard remote-agent project
Drop the dedicated test_audio_e2e Makefile target and the separate
remote-agent-audio Playwright project. The remote-agent project now
matches every ra-*.spec.ts under e2e/remote-agent, so make test_e2e
runs ra-audio.spec.ts alongside ra-all.spec.ts in the same worker.
* Fall back to bare-track MediaStream when ontrack streams[] is empty
Hit a state where pc.getReceivers() showed live video and audio tracks
but useRTCStore.mediaStream stayed undefined — the SDP answer arrived
without a=msid, so event.streams[0] was undefined and
setMediaStream(undefined) left the store empty even though RTP was
flowing. Only a hard reload recovered.
Now: when the event carries a stream, use it as before. When it doesn't,
get-or-create a MediaStream and append the track. Re-using the existing
store value across both ontrack invocations keeps audio + video on the
same MediaStream so the autoplay/video pipeline downstream is unchanged.
* Close peer connection before reload-on-toggle
Firefox's soft reload doesn't always tear down the RTCPeerConnection,
which leaves the post-reload page in a half-renegotiated state: tracks
arrive on receivers but never attach to a MediaStream, so video stays
stuck on "Loading…" (or the page falls back to the pre-connect blue
background) until a hard refresh. Closing the PC explicitly before
reload guarantees a clean start.
* Aggregate ontrack into one canonical MediaStream
The answer SDP from pion omits a=msid for the audio track in some
configurations (visible on Firefox: video keeps its msid, audio doesn't).
The previous handler called setMediaStream(event.streams[0]) on each
ontrack, so:
1. video ontrack → setMediaStream(streamA) [has video]
2. audio ontrack → setMediaStream(streamB) [synthetic, audio only]
streamB replaces streamA, video disappears. Hard refresh only "fixed" it
incidentally — the same SDP would break the next negotiation too.
Now: ignore event.streams[0], maintain one canonical MediaStream in the
store, and addTrack into it on every ontrack. Browsers render tracks
added to a live MediaStream that's already attached to srcObject, so
both audio and video stay attached regardless of which order ontrack
fires or whether the SDP carried msid.
* Only render <audio> element when device audio is enabled
The backend keeps the m=audio section in the SDP even when audio is
disabled (just inactive direction), so Firefox still attaches a muted
audio track to the MediaStream. The autoplay <audio> element then
triggers Firefox's "block audio" policy on a stream that will never
actually play any sound.
Fetch getAudioConfig once the RPC channel is up, then conditionally
render the <audio> element. No autoplay prompt when audio is off.
* Drop stale mediaStream on reconnect; release audio capture when owner session ends
* Back off briefly on idle ALSA reads to avoid CPU spin
* Re-wire <audio> when track arrives late; gate VideoStart to first session
195 lines
4.7 KiB
Go
195 lines
4.7 KiB
Go
package kvm
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"os"
|
|
"path/filepath"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/jetkvm/kvm/internal/audio"
|
|
"github.com/pion/webrtc/v4"
|
|
"github.com/pion/webrtc/v4/pkg/media"
|
|
)
|
|
|
|
var (
|
|
audioCancel context.CancelFunc
|
|
audioStopped chan struct{}
|
|
audioTrack *webrtc.TrackLocalStaticSample
|
|
audioMu sync.Mutex
|
|
)
|
|
|
|
func startAudio(track *webrtc.TrackLocalStaticSample) {
|
|
audioMu.Lock()
|
|
defer audioMu.Unlock()
|
|
stopAudioLocked()
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
audioCancel = cancel
|
|
audioStopped = make(chan struct{})
|
|
audioTrack = track
|
|
|
|
go runAudioCapture(ctx, track, audioStopped)
|
|
}
|
|
|
|
func stopAudio() {
|
|
audioMu.Lock()
|
|
defer audioMu.Unlock()
|
|
stopAudioLocked()
|
|
}
|
|
|
|
// stopAudioIfOwner stops the audio capture only if it is currently bound to
|
|
// track. Used on session teardown so capture doesn't keep writing samples to
|
|
// a track whose peer connection has closed.
|
|
func stopAudioIfOwner(track *webrtc.TrackLocalStaticSample) {
|
|
audioMu.Lock()
|
|
defer audioMu.Unlock()
|
|
if audioTrack != track {
|
|
return
|
|
}
|
|
stopAudioLocked()
|
|
}
|
|
|
|
func stopAudioLocked() {
|
|
if audioCancel == nil {
|
|
return
|
|
}
|
|
audioCancel()
|
|
<-audioStopped
|
|
audioCancel = nil
|
|
audioStopped = nil
|
|
audioTrack = nil
|
|
}
|
|
|
|
// reopenThreshold is the number of consecutive non-idle read errors that
|
|
// triggers a close+reopen of the ALSA handle. The C-side already recovers
|
|
// EPIPE/ESTRPIPE; errors that surface here (EBADFD, ENODEV, …) usually mean
|
|
// the handle is dead — typically a USB gadget rebuild or host reattach.
|
|
const reopenThreshold = 5
|
|
|
|
func runAudioCapture(ctx context.Context, track *webrtc.TrackLocalStaticSample, stopped chan<- struct{}) {
|
|
defer close(stopped)
|
|
|
|
codec := audio.CodecPCMU
|
|
if strings.EqualFold(track.Codec().MimeType, webrtc.MimeTypeG722) {
|
|
codec = audio.CodecG722
|
|
}
|
|
|
|
capture, err := openCaptureWithBackoff(ctx)
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer func() { capture.Close() }()
|
|
|
|
audioLogger.Info().Str("codec", codec.String()).Msg("audio capture started")
|
|
defer audioLogger.Info().Msg("audio capture stopped")
|
|
|
|
sample := media.Sample{Duration: 20 * time.Millisecond}
|
|
consecutiveErrors := 0
|
|
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
default:
|
|
}
|
|
|
|
payload, err := capture.ReadEncoded(codec)
|
|
if err != nil {
|
|
if errors.Is(err, audio.ErrNoAudioData) {
|
|
// Partial period or idle ALSA — back off ~half a frame so we
|
|
// don't spin while the buffer fills.
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-time.After(10 * time.Millisecond):
|
|
}
|
|
continue
|
|
}
|
|
consecutiveErrors++
|
|
audioLogger.Warn().Err(err).Int("errs", consecutiveErrors).Msg("audio capture read failed")
|
|
if consecutiveErrors >= reopenThreshold {
|
|
capture.Close()
|
|
next, err := openCaptureWithBackoff(ctx)
|
|
if err != nil {
|
|
return
|
|
}
|
|
capture = next
|
|
consecutiveErrors = 0
|
|
continue
|
|
}
|
|
time.Sleep(100 * time.Millisecond)
|
|
continue
|
|
}
|
|
|
|
consecutiveErrors = 0
|
|
if len(payload) == 0 {
|
|
continue
|
|
}
|
|
|
|
sample.Data = payload
|
|
if err := track.WriteSample(sample); err != nil {
|
|
audioLogger.Warn().Err(err).Msg("audio sample write failed")
|
|
time.Sleep(100 * time.Millisecond)
|
|
}
|
|
}
|
|
}
|
|
|
|
// openCaptureWithBackoff opens the ALSA capture device, retrying with
|
|
// exponential backoff (capped at 2 s) until success or ctx cancellation.
|
|
// Re-resolves the card on every attempt so a USB re-enumeration that hands
|
|
// the gadget a new card number is picked up automatically.
|
|
func openCaptureWithBackoff(ctx context.Context) (*audio.ALSACapture, error) {
|
|
backoff := 100 * time.Millisecond
|
|
for {
|
|
device := alsaCaptureDevice()
|
|
capture, err := audio.OpenALSACapture(device)
|
|
if err == nil {
|
|
audioLogger.Info().Str("device", device).Msg("audio capture opened")
|
|
return capture, nil
|
|
}
|
|
audioLogger.Warn().Err(err).Str("device", device).Dur("retry_in", backoff).Msg("audio capture open failed")
|
|
select {
|
|
case <-ctx.Done():
|
|
return nil, ctx.Err()
|
|
case <-time.After(backoff):
|
|
}
|
|
if backoff *= 2; backoff > 2*time.Second {
|
|
backoff = 2 * time.Second
|
|
}
|
|
}
|
|
}
|
|
|
|
// alsaCaptureDevice returns the ALSA device for the UAC1 gadget card.
|
|
func alsaCaptureDevice() string {
|
|
if card, ok := findALSACard("UAC1Gadget"); ok {
|
|
return "hw:" + strconv.Itoa(card) + ",0"
|
|
}
|
|
return "hw:1,0"
|
|
}
|
|
|
|
func findALSACard(cardID string) (int, bool) {
|
|
entries, err := os.ReadDir("/sys/class/sound")
|
|
if err != nil {
|
|
return 0, false
|
|
}
|
|
|
|
for _, entry := range entries {
|
|
name := entry.Name()
|
|
if !strings.HasPrefix(name, "card") {
|
|
continue
|
|
}
|
|
id, err := os.ReadFile(filepath.Join("/sys/class/sound", name, "id"))
|
|
if err != nil || strings.TrimSpace(string(id)) != cardID {
|
|
continue
|
|
}
|
|
if card, err := strconv.Atoi(strings.TrimPrefix(name, "card")); err == nil {
|
|
return card, true
|
|
}
|
|
}
|
|
return 0, false
|
|
}
|