mirror of
https://github.com/jetkvm/kvm.git
synced 2026-06-20 05:24:32 +00:00
dev
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Adam Shiervani
|
d5d473bdbd | Fix e2e flakyness | ||
|
Adam Shiervani
|
90750bf609 |
Experimental Audio Support (#1475)
* WIP: USB audio capture via UAC1 gadget with G.722 + PCMU encoding
Adds ALSA capture from the USB UAC1 gadget, G.722/PCMU encoding, a WebRTC
audio track, and an e2e remote-agent flow that plays a tone on the remote
host and verifies it reaches the browser.
Snapshot of codex-driven implementation before simplification.
* Simplify USB audio path: drop residual scaffolding from long session
Backend:
- audio.go: drop source rotation + "no-data reopen with next device" loop.
One source (UAC1Gadget; falls back to hw:1,0 only if sysfs lookup fails).
- internal/audio: remove unused Reader interface and unavailableCapture stub.
Stub now returns the concrete type with an error.
- webrtc.go: inline single-use resolveAudioCodec helper; DRY video/audio RTCP
drain into drainRTCP; fold startSessionAudio into the connect callback.
Frontend:
- devices.$id.tsx: drop remoteMediaStreamRef track-merging. Backend tracks
share stream ID "kvm", so pion delivers them in one MediaStream — just
assign event.streams[0].
- WebRTCVideo.tsx: replace dynamic per-track <audio> creation + ref array
with a single hidden <audio> bound to mediaStream.
Remote agent:
- Drop PipeWire/wpctl detection path; plughw: works directly.
- Drop killStaleAudioToneProcesses pkill workaround; the (cmd, cancel, done)
trio collapses to a single *exec.Cmd field with Start/Kill/Wait.
E2E:
- ra-audio.spec.ts: drop attachAudioDiagnostics scaffold and openReadyPage
duplicate. Spec is now linear: setup → wait for track → diff stats → tone.
Net: ~355 LOC removed.
* Fix onSessionConnected race; replace onFirst/CurrentSessionConnected split
The previous codex split routed audio start through onCurrentSessionConnected,
gated on session == currentSession. But currentSession is assigned by the
caller (web.go, cloud.go) AFTER ExchangeOffer returns, while
OnICEConnectionStateChange can fire from inside ExchangeOffer or shortly
after — racing the assignment. When the race hits, the equality check fails,
the callback is skipped, and audio never starts.
Pass session into the callback directly so the per-session setup uses the
session in hand, not whatever currentSession happens to point to at that
instant. Keep stopVideoSleepModeTicker on the count-edge (still only on
first-session) and let onSessionConnected handle the rest unconditionally.
* Pin WebRTC playout-delay to zero so receivers don't ratchet under stress
Chrome's adaptive receive-side jitter buffer grows under stress (e.g.
playing a video on the controlled machine) and does not reliably shrink
back; the Connection Stats "Playback Delay" graph used to climb to
~300 ms and stay there until the page was reloaded.
The trigger is the USB UAC1 audio path, not video motion per se — once
real audio starts flowing, Chrome's AV-sync layer pulls the video
jitter buffer up to whatever the audio path settles at, and the ratchet
locks in. Receiver-side hints (jitterBufferTarget, playoutDelayHint,
setMinimumJitterBufferDelay) cap the steady state but don't release a
buffer that has already grown.
Fix: register the WebRTC playout-delay RTP header extension on both
audio and video and stamp min=max=0 on every outgoing packet via a
pion interceptor. Chrome treats this as an authoritative override of
its adaptive logic and keeps both buffers at the decoder floor through
and after stress, with no peer-connection rebuild needed.
Test: drive the host display with a real audio+video file via
gst-launch playbin (audio routed through PipeWire to the USB UAC1
sink) and assert receive-side video delay stays bounded both during
and after playback.
* Simplify audio capture loop and ALSA reader
- Drop short-read zero-fill in ALSA reader; return ErrNoAudioData so the
capture loop emits no frame for the cycle instead of half-silent audio.
- Replace ErrNoAudioData = io.ErrNoProgress (wrong semantic) with a domain
sentinel and remove the unused idleReads debug counter.
- Encoders sum all source samples before one divide — better precision,
fewer ops; clampS16 and sampleS16 helpers gone.
- Resolve audio codec inline in runAudioCapture; drop the audioCodecForTrack
wrapper. Caller checks AudioTrack != nil so startAudio no longer accepts
nil as a stop signal.
- Use C.GoString instead of hand-rolled cString helper.
- Add a why-comment on the separate <audio> element (video stays muted).
* Trim playoutdelay per-symbol comments
Keep the package-level "why" (Chrome's one-way jitter buffer); drop
restate-the-signature comments on Factory, NewFactory, NewInterceptor,
and BindLocalStream.
* e2e: extract shared audio helpers and drop dev-only override
- Lift ensureNoPasswordViaAPI and waitForAudioStream into helpers.ts (the
audio spec was inlining both, the latter as a copy of waitForVideoStream).
- ra-audio.spec.ts shrinks from 78 to 55 lines.
- Remove the JETKVM_AUDIO_DEVICE override in the remote agent: it fabricated
an AudioDeviceInfo with is_jetkvm=true regardless of what device the env
var pointed at, silently lying to the spec's assertion. Audio device
discovery via aplay + /proc/asound/.../usbid is reliable; if no JetKVM
device is present the spec already skips.
* Reopen ALSA capture on persistent read errors
The C-side recovers EPIPE/ESTRPIPE via snd_pcm_recover; the errors that
surface to Go (EBADFD, ENODEV, …) usually mean the handle is dead —
typically a USB gadget rebuild or host reattach mid-session, which used
to leave audio silent until the session disconnected.
After 5 consecutive non-idle read errors, close and reopen the capture
with exponential backoff (100 ms → 2 s cap). Initial open uses the same
helper so we keep retrying instead of giving up if the gadget isn't ready
yet. Re-resolves the card each attempt so a USB re-enumeration that
shifts the card number is picked up automatically.
* Add Audio settings page with Enable Audio toggle (experimental)
Audio is opt-in via device config. New Audio nav entry in Settings sits
next to Video, with a single "Enable Audio" item marked Experimental
(mirrors HTTPS Mode in Access).
Backend:
- Config.AudioEnabled (default false), persisted to /userdata/kvm_config.json
- getAudioConfig / setAudioConfig JSON-RPC handlers
- webrtc.go: extract attachAudioTrack helper; skip track creation when
disabled or when the offer advertises no supported codec. The SDP
answer leaves the audio m-line inactive, so flipping the toggle
requires a fresh connection (page reload).
Frontend:
- New devices.$id.settings.audio.tsx — fetches state via getAudioConfig,
saves via setAudioConfig, optimistic UI with rollback on error.
- devices.$id.tsx always offers audio in the SDP; backend decides.
- en.json + 13 locale files: 5 keys each (audio_*, settings_audio) with
proper translations honoring per-language formality.
E2E:
- ra-audio.spec.ts: connect, enable via RPC, reload, verify audio energy.
Restores disabled state in a finally block so other specs aren't
affected. 9 s on kvm-2 + .180.
* Reload page after enabling audio so the browser prompts for autoplay
Re-negotiation only happens on a fresh WebRTC session, and the autoplay
overlay needs a user gesture to play the new audio track. A simple reload
covers both — the toggle's user click acts as the gesture, the new offer
includes audio, and the overlay surfaces normally.
Disable path is unchanged (audio stops naturally on the next connect).
* Reload page on disable too so audio stops immediately
* Trim audio_* copy across all 14 locales
Page header and item description previously said roughly the same thing
in long form. Now: page-level describes the topic ("Stream audio from
the host to your browser"); item-level is a terse one-liner ("Stream
HDMI audio from the host."). Drops the "Requires a fresh connection"
clause — the page auto-reloads on toggle, so it's no longer accurate.
Per-language tone follows I18N_BEST_PRACTICES.md: formal Sie/vous/usted/
вы/chi (de/fr/es/ru/cy), informal du (sv/nb/da), polite です/ます (ja),
infinitive (it), European Portuguese (pt).
* Fold audio e2e into the standard remote-agent project
Drop the dedicated test_audio_e2e Makefile target and the separate
remote-agent-audio Playwright project. The remote-agent project now
matches every ra-*.spec.ts under e2e/remote-agent, so make test_e2e
runs ra-audio.spec.ts alongside ra-all.spec.ts in the same worker.
* Fall back to bare-track MediaStream when ontrack streams[] is empty
Hit a state where pc.getReceivers() showed live video and audio tracks
but useRTCStore.mediaStream stayed undefined — the SDP answer arrived
without a=msid, so event.streams[0] was undefined and
setMediaStream(undefined) left the store empty even though RTP was
flowing. Only a hard reload recovered.
Now: when the event carries a stream, use it as before. When it doesn't,
get-or-create a MediaStream and append the track. Re-using the existing
store value across both ontrack invocations keeps audio + video on the
same MediaStream so the autoplay/video pipeline downstream is unchanged.
* Close peer connection before reload-on-toggle
Firefox's soft reload doesn't always tear down the RTCPeerConnection,
which leaves the post-reload page in a half-renegotiated state: tracks
arrive on receivers but never attach to a MediaStream, so video stays
stuck on "Loading…" (or the page falls back to the pre-connect blue
background) until a hard refresh. Closing the PC explicitly before
reload guarantees a clean start.
* Aggregate ontrack into one canonical MediaStream
The answer SDP from pion omits a=msid for the audio track in some
configurations (visible on Firefox: video keeps its msid, audio doesn't).
The previous handler called setMediaStream(event.streams[0]) on each
ontrack, so:
1. video ontrack → setMediaStream(streamA) [has video]
2. audio ontrack → setMediaStream(streamB) [synthetic, audio only]
streamB replaces streamA, video disappears. Hard refresh only "fixed" it
incidentally — the same SDP would break the next negotiation too.
Now: ignore event.streams[0], maintain one canonical MediaStream in the
store, and addTrack into it on every ontrack. Browsers render tracks
added to a live MediaStream that's already attached to srcObject, so
both audio and video stay attached regardless of which order ontrack
fires or whether the SDP carried msid.
* Only render <audio> element when device audio is enabled
The backend keeps the m=audio section in the SDP even when audio is
disabled (just inactive direction), so Firefox still attaches a muted
audio track to the MediaStream. The autoplay <audio> element then
triggers Firefox's "block audio" policy on a stream that will never
actually play any sound.
Fetch getAudioConfig once the RPC channel is up, then conditionally
render the <audio> element. No autoplay prompt when audio is off.
* Drop stale mediaStream on reconnect; release audio capture when owner session ends
* Back off briefly on idle ALSA reads to avoid CPU spin
* Re-wire <audio> when track arrives late; gate VideoStart to first session
|
||
|
Adam Shiervani
|
c08f14ff3f |
fix: send mouse button state changes via reliable WebRTC channel (#695) (#1338)
* fix: send mouse button state changes via reliable WebRTC channel (#695) When holding one mouse button and pressing another without moving the mouse, only pointerdown/pointerup events fire (no mousemove to self-correct). These button-only state changes were sent via the unreliable WebRTC data channel (maxRetransmits: 0), and lost packets were never recovered. Changes: - useHidRpc.ts: Track last button state and send button changes via the reliable channel. Movement-only events continue using the unreliable channel for low latency, since lost movement packets self-correct via subsequent mousemove events. - e2e/remote-agent/main.go: Fix omitempty on InputEvent.Value so button release events (value=0) are included in JSON responses. - ra-all.spec.ts: Add E2E test that holds left mouse button, presses and releases right mouse button, and verifies all 4 button events arrive on the remote host (20 iterations). * fix: rebuild remote agent when source changes to prevent stale deploys ensureDeployed() skipped rebuild and redeploy when the agent was already running, causing the omitempty fix on InputEvent.Value to never reach the remote host. Now compares source mtime against binary mtime and forces redeploy when a rebuild occurs. |
||
|
Adam Shiervani
|
203c6ae6fd |
fix: recover HID chardev after DWC3 rebind race on RV1106 (#1366)
* fix: reset USB gadget when virtual media unmount fails with EBUSY (#834) When unmountImageLocked() gets EBUSY from the kernel (host OS still accessing the virtual disk via PREVENT MEDIUM REMOVAL), fall back to gadget.RebindUsb(true) to force-disconnect the host, then retry the unmount. Uses RebindUsb directly instead of UpdateGadgetConfig to avoid hitting the same EBUSY when writing configfs attributes before rebind. After rebind, properly reopen keyboard HID file (ResetHIDFiles + sleep + OpenKeyboardHidFile) matching the pattern in setMassStorageMode(). Also propagate unmount errors to RPC callers and only clear currentVirtualMediaState after the unmount actually succeeds. Adds E2E test that mounts an ISO on the remote host to trigger PREVENT MEDIUM REMOVAL, then verifies unmount succeeds and keyboard recovers. * fix: recover HID chardev after DWC3 rebind race on RV1106 The DWC3 USB controller on the RV1106 has a race condition where rapid unbind→bind of the UDC can permanently corrupt HID chardev state — /dev/hidg0 returns ENXIO even though the device node exists and the UDC shows "configured". This can be triggered by UpdateGadgetConfig's transaction rebind and by host-initiated USB device resets during mass storage media changes. Three-part fix: 1. rebindUsb(): after binding, verify /dev/hidg0 is openable. If not, unbind again with a 100ms pause for kernel cleanup, then rebind. 2. setMassStorageMode(): pre-set recovery timer before UpdateGadgetConfig to prevent the poller from interfering. After the 1s sleep, if OpenKeyboardHidFile fails, do a corrective RebindUsb + retry. 3. checkUSBState() poller: when a state transition occurs and OpenKeyboardHidFile fails, trigger a corrective rebind to recover from host-initiated USB resets that corrupt the chardev. * fix: suppress USB recovery poller before rebind in unmount and mode-change paths The auto-recovery poller could see transient "not attached" UDC state during RebindUsb and trigger a competing rebind, corrupting HID chardev state. Add setUSBRecoveryTimer calls before the rebind in unmountImageLocked and before the corrective rebind in setMassStorageMode. * test: replace blind sleep with two-phase wait in factory-reset e2e test Wait for device to become unreachable before polling for it to come back, preventing false passes from stale pre-reset responses. * refactor: simplify branch — extract helpers, remove duplication, fix flaky tests - Extract rebindAndRecoverHID() in Go to deduplicate USB recovery sequences - Remove redundant setUSBRecoveryTimer() call after UpdateGadgetConfig() - Extract waitForKeyboardReady() helper replacing 5 duplicate retry loops - Consolidate 3 duplicate remoteExec definitions into single remoteHostExec() - Use shared SSH_OPTS from helpers.ts instead of hardcoded SSH options - Fix remote agent omitempty on mouse X/Y causing undefined in TypeScript - Poll keys-down state in disconnect test to avoid race condition * fix: remove dead IsHidgChardevHealthy export, reset HID files before rebind - Remove unused exported IsHidgChardevHealthy wrapper (only the unexported isHidgChardevHealthy is called, inside rebindUsb) - Move ResetHIDFiles() before RebindUsb in checkUSBState so stale file handles are closed even if the rebind fails — prevents silent mouse write failures on dead inodes after a successful unbind + failed bind |
||
|
Adam Shiervani
|
cf7215411d |
feat: add horizontal mouse wheel scrolling support (#415) (#1358)
* fix: add horizontal mouse wheel (AC Pan) scroll support (#415) - HID descriptors: add AC Pan (Usage 0x0238, Consumer Page) to both absolute and relative mouse descriptors for horizontal scroll - Backend: extend AbsMouseWheelReport to accept wheelX, add RelMouseWheelReport with both axes, update report_length - RPC: add wheelX parameter to wheelReport binding - Frontend: read deltaX in wheel handler with same clamping/inversion and throttling as vertical scroll - E2E: add wheel scroll test verifying both vertical (REL_WHEEL) and horizontal (REL_HWHEEL) events reach the remote host * style: fix goimports alignment in RelMouseReport * fix: don't negate horizontal scroll direction in wheelReport The clampWheel helper was negating the result for both axes, but only vertical scrolling needs inversion (browser deltaY and HID Wheel use opposite sign conventions). Horizontal scrolling (deltaX / AC Pan) shares the same convention (positive = right), so negation reversed the direction on the target machine. * fix: wire RelMouseWheelReport into RPC and add wheel scroll e2e tests rpcWheelReport only called AbsMouseWheelReport, so wheel scrolling was silently broken in relative-only mouse mode. Now calls both Abs and Rel wheel report methods (each guards on its own enabledDevices flag). Adds e2e tests for vertical/horizontal wheel scroll in default mode and relative-only mode. Bumps beforeAll waitForInputDevices timeout to 30s and keyboard LED test expectKeyPress timeouts to 5s to reduce flakiness. |
||
|
Adam Shiervani
|
15dc380062 |
fix: auto-recover USB gadget when host power-cycles (#1297)
* fix: auto-recover USB gadget when host reconnects (#128) When the USB host reboots or disconnects, the UDC state becomes "not attached" and never recovers. Add automatic recovery that detects this state and rebinds the USB gadget, with rate limiting to avoid thrashing. Also refactor keyboard HID file handling to support force-reopen after rebind. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: move USB recovery logic to internal/usbgadget package Extract ShouldAttemptUSBRecovery and its retry interval constant into the internal/usbgadget package so the logic is testable without importing the top-level kvm package. Reset the recovery timer in updateUsbRelatedConfig to prevent auto-recovery from interfering during the host's USB re-enumeration window after a deliberate config change. Made-with: Cursor * feat(e2e): add JSON-RPC data channel support to test hooks Expose the WebRTC RPC data channel through test hooks and add a sendJsonRpc helper that sends a JSON-RPC request over the channel and resolves via callback with timeout handling. This enables e2e tests to invoke backend RPC methods directly. Made-with: Cursor * refactor(e2e): restructure tests with remote-agent suite Move device-level e2e tests (config-reset, EDID, HTTPS, HDMI sleep, LED, mouse, USB attach timing, USB device) into a consolidated remote-agent suite that runs through a Go-based agent binary. Add a separate Playwright project for remote-agent tests. - Remove standalone spec files now covered by ra-all.spec.ts - Rename login-rate-limit to zz-login-rate-limit so it runs last (avoids needing a post-test reboot to clear rate-limit state) - Reorder welcome-password tests so validation runs first, reusing the onboarding state and saving an SSH reset cycle Made-with: Cursor * refactor(e2e): clean up test helpers and reduce duplication Consolidate all test helpers into a single helpers.ts file, removing the separate ota-helpers.ts. This gives every test file a single import source and eliminates duplicated code across the test suite. Key changes: - Merge ota-helpers.ts into helpers.ts (mock server, binary deployment, device config, env var validation, triggerUpdate, withTempSignature) - Remove duplicated rpc/restartAppViaSSH/waitForDeviceReady functions from ra-all.spec.ts in favour of shared imports - Extract loginAndOpenSettings helper in settings-local-auth tests - Extract getOTAEnvVars, toPreReleaseVersion, triggerUpdate, and withTempSignature to reduce boilerplate across OTA tests - Remove unused verifyMouseWorks function and dead variables - Strip redundant JSDoc that just restated type signatures - Remove duplicated per-project use config from playwright.config.ts (already inherited from top-level) - Convert dynamic imports in sshExec to top-level imports Made-with: Cursor * refactor(e2e): move binary deployment into Playwright globalSetup Replace the shell-script deployment logic with Playwright's globalSetup/globalTeardown hooks. When BASELINE_BINARY_PATH is set, globalSetup deploys the binary, resets device config, reboots, and captures pre-test logs. globalTeardown captures post-test logs. This keeps the deployment lifecycle inside Playwright where it belongs, and reduces test_core_e2e.sh to a thin wrapper that sets env vars. Made-with: Cursor * fix: retry HID file reopen after USB gadget rebind After rebinding the DWC3 USB controller, the kernel needs a moment to create the /dev/hidg* device nodes. The previous code attempted to reopen the keyboard HID file immediately after rebind, which raced with the kernel and failed with "no such device or address". Add a retry loop (up to 10 attempts, 200ms apart) to wait for the device nodes to appear before reopening the keyboard HID file. Made-with: Cursor * fix: harden USB gadget recovery after UDC unbind Reset stale HID gadget handles after rebind, suppress transient HID-open errors during detach windows, and fall back to full gadget reconfiguration when simple UDC rebind does not restore keyboard HID promptly. Strengthen the remote-agent USB recovery E2E to verify both keyboard and mouse input recover after unbind with retry tolerance for host-side input node churn. Made-with: Cursor * refactor: simplify USB HID error handling and reduce hot-path overhead - Use errors.Is with syscall.Errno instead of string matching in IsHIDTemporarilyUnavailableError (robust, zero-alloc) - Cache USB state in usbReadyForHidReports instead of reading sysfs on every HID report - Extract rpcHidReport wrapper to deduplicate 5 rpc*Report functions - Fix openWithTimeout goroutine/fd leak on timeout - Add USBStateNotAttached/USBStateUnknown constants, replace literals - Deduplicate discoverJetKVMDevices by delegating to listInputDevices Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: run remote-agent e2e tests when JETKVM_REMOTE_HOST is provided Made-with: Cursor --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |