mirror of
https://github.com/TelegramMessenger/tgcalls.git
synced 2026-11-18 21:35:28 +00:00
dev/android
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
60139c276a |
group: ReferenceImpl asks the SFU for video the way CustomImpl does
A device log (2026-09-22) showed a tile frozen after its endpoint was requested again: two endpoints received zero RTP for 9 s and 48 s while every client-side step (transceiver, receive stream, demuxer entries, sink proxy, constraints resent 3-4 times over a healthy data channel) matched the earlier requests that worked. The fix for unsignaled receive streams ( |
||
|
|
f359144db7 |
group: ReferenceImpl never lets WebRTC create unsignaled receive streams
Two production defects with one cause: a packet the network thread has
already handed to the worker when the worker tears down a receive
stream is received as an unknown SSRC, and WebRTC creates a default
("unsignaled") receive stream for it on a channel whose remote content
never changes again. That stream is never reset and keeps the SSRC's
binding in the Call's RtpDemuxer, so the m-line later negotiated for
the SSRC logs "Sink could not be added" and never receives a packet.
Video (device log 2026-09-18): after the app dropped all video
endpoints and requested them again on the same PeerConnection, one tile
stayed frozen while every decoded frame logged
"VideoReceiveStreamInterface not connected to a VideoSink". The factory
now gets its own FieldTrialsView (GRFieldTrials) that enables
WebRTC-Video-DiscardPacketsWithUnknownSsrc and resolves everything else
through tgcalls' shared FieldTrialBasedConfig; before, the engine
inherited the trial only if a CustomImpl call had run earlier in the
process.
Audio (found in the CLI while reproducing the above): the same race at
the first discovery renegotiation recreated an unsignaled stream on the
mid-0 catch-all right after WebRTC reset it. The participant stayed
audible through that stream, but the level sink on the dedicated
receiver never fired, so peers never saw them as speaking (2 of 6 level
sinks dead in a 3-participant run). The outgoing audio transceiver is
now sendonly (answered recvonly), the GRAudioFrameTransformer catch-all
is gone, discovery goes only through the un-demuxable-packet tap, and
the PeerConnection opts into the new vendored-webrtc seam
RTCConfiguration::disable_payload_type_demuxing. The seam is
load-bearing: without it the lone recvonly audio m-line becomes the
payload-type catch-all and a late unmuter is swallowed there
(--unmute-after scored 0/2). First remote audio now arrives ~280 ms
after the first packet (debounce plus one offer/answer), as with
CustomImpl.
CLI: --video-rerequest drops and re-requests every video endpoint five
times under a 150 ms SFU downlink delay and requires every sink to
receive frames past the count taken just before the re-request
(reproduces the frozen tile with --reference-participants 6: 1/30
stalled on the unfixed engine, both runs). Every group run now also
prints "Audio levels heard: X/Y pairs" (every peer must report a real
level for every unmuted sender) and "Unsignaled audio: N stream(s)
(must be 0)" (engine log scanned for the stream WebRTC creates on its
own; 6 for 3 reference participants before). Full sweep green: audio
3 and 6 reference participants, late unmute (reference and mixed),
mixed e2e with video, reference e2e, custom-only, early video request,
capture feeds, 6-participant re-request, group-churn, bandwidth
step-down scenarios.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
ffed33c535 |
group: release _isRenegotiating when a renegotiation cannot proceed
A failed renegotiation CreateOffer or a failed buildRemoteAnswer returned
without completing the renegotiation, so _isRenegotiating stayed set and
every later renegotiation was deferred forever: a participant joining
afterwards never got a receive transceiver and stayed inaudible. Both
paths now post onRenegotiationComplete(), like the SLD/SRD failure paths.
Reviewed GroupInstanceReferenceImpl for the 18/19 start-glare class while
here: not affected (single offerer, locally built answer, serialised
renegotiation, camera swaps the track on a pre-created transceiver).
tools/cli: TGCALLS_CLI_KEEP_LOGS=1 keeps the per-participant group logs
of a passing run (they are written at stop and were always unlinked).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
44cb173060 |
group: stop the outgoing audio stream when ReferenceImpl is muted
GroupInstanceReferenceImpl muted by disabling the outgoing audio track only.
In WebRTC that reaches ChannelSend::SetInputMute, which zeroes the samples
but keeps encoding and sending, so a muted participant still emitted a full
Opus stream (~50 packets/s at level 0): invisible to every level-based check
and to the peers, but paid for by the SFU and every receiver. What actually
starts and stops the AudioSendStream is the sender's encodings[0].active
(WebRtcAudioSendStream::UpdateSendState); this vendored WebRTC starts the
stream even with no source, so detaching the track would not have helped.
GroupInstanceCustomImpl was never affected: its mute disables the channel.
applyOutgoingAudioMuteState() now toggles that flag through
GetParameters/SetParameters with no renegotiation, keeps the track disabled
so the AGC/level path still sees the mute, and applies the ADM microphone
mute as CustomImpl's onUpdatedIsMuted does - on iOS that is the system mute,
which also drives the muted-speech hint and which the reference engine had
never engaged. It also runs from start(), so the stream is stopped from the
first negotiation via the sender's init parameters. Stopping the only send
stream makes AudioState stop ADM recording, as CustomImpl's muted state
already does; the muted-speech detector lives in the audio unit, not the
recording path, so it is unaffected.
CLI: the Go SFU counts inbound audio RTP per SSRC (GoSfu_QuerySsrcPackets),
read just before teardown, and group validation adds "Muted audio leaks":
every participant that stayed muted must have sent none. A muted reference
participant sent 456 packets in a 10 s run before this change and 0 after;
the late-unmute regressions still score 2/2 with the reference participant
as the late unmuter, so unmute restarts the stream.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
955b30e30c |
group: discover audio SSRCs below the demuxer so a mid-call unmute is heard
A participant who unmuted after the call had settled was inaudible to
GroupInstanceReferenceImpl for the rest of the call, and rejoining only
recovered whoever happened to be sending during the ~250 ms before the first
renegotiation — so rejoining a quiet room could give no audio at all.
The engine discovered audio SSRCs from a frame transformer on mid=0, which only
ever sees packets WebRTC was willing to route to the unsignaled catch-all.
SdpOfferAnswerHandler::UpdatePayloadTypeDemuxingState stops being willing as soon
as two *receiving* audio m-lines in a BUNDLE group advertise the same payload
type, and it also calls ResetUnsignaledRecvStream(). Every audio m-line here
advertises Opus 111 (sendrecv mid=0 plus one recvonly per remote SSRC), so the
FIRST discovery renegotiation disabled payload-type demuxing permanently. After
that an unknown SSRC had no MID extension (buildRemoteAnswer strips it), no SSRC
binding and no payload type to fall back on: RtpDemuxer dropped it, no receive
stream was created, and therefore no frame transformer could ever see it.
GroupInstanceCustomImpl was never affected — it receives all remote audio on one
channel, never adds a second m-line, and discovers from raw RTP in
GroupNetworkManager.
Discovery now happens where the packet is still visible: the ICE transport.
MtProtoIceTransport gains an optional IncomingPacketObserver, and its mtproto
encryption becomes optional — with no key it is a pure pass-through (packets and
flags untouched, so DTLS-SRTP above is unaffected) whose only job is the
observer. The group engine injects it that way and reads the SSRC from the
cleartext RTP header; SRTP encrypts the payload, not the header. With a key the
observer runs on the decrypted packet instead, so observation is a byproduct of
the decryption pass either way.
While encryption is on and the sender is still unknown the tap keeps reporting,
mirroring GRAudioFrameTransformer: a description response that omits the SSRC
would otherwise leave that participant permanently undecryptable, and past the
first renegotiation nothing else can ask again. The de-duplication set degrades
to reporting everything when full rather than to reporting nothing — going deaf
to new speakers is the failure this tap exists to remove.
Also register the LogSinkImpl. The constructor built one and the destructor
removed it, but nothing ever called AddLogToStream, so every reference-engine
call wrote a zero-byte log and none of this was diagnosable from a bug report.
An app-side roster (addSsrcs from the participant list) was implemented first and
rejected: this engine adds one recvonly m-line per SSRC and a voice chat's roster
runs to thousands.
CLI: --unmute-after unmutes mid-call while signalling nothing to the peers, and
validation requires every peer to report a real level for the late SSRC. It
scored 1/2 before this change and 2/2 after; the all-custom group and the same
late unmute with no prior renegotiation pass either way, which is what pins the
cause to the renegotiation rather than to the lateness.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
b99dfc0dd6 |
group: ReferenceImpl sends the app's camera video and owns its own endpoint
The iOS wrapper hands the camera over as a VideoCaptureInterface
(descriptor.videoCapture when the camera is on at join, setVideoCapture for
requestVideo:/disableVideo:) and never calls setVideoSource; only the CLI
harness sets descriptor.getVideoSource. GroupInstanceReferenceImpl's
setVideoCapture was an empty stub and its ctor ignored descriptor.videoCapture,
so in a real group call the camera ran, the server marked the participant
video-on, and no frame was ever encoded (no SetVideoSend, the sendonly
transceiver stayed trackless). Route both wrapper paths through the shared
videoCaptureToGetVideoSource (moved out of GroupInstanceCustomImpl into
VideoCaptureInterfaceImpl so the engines cannot drift) and have setVideoSource
store/clear _getVideoSource, so a camera switched off before the join
handshake completes is not re-attached by addRemoteIceCandidates.
The app also hands the engine its full roster, itself included, both in
setRequestedVideoChannels and in addIncomingVideoOutput for the local tile,
and relies on the engine to know which entry is "me". Learn the own endpoint
from the join response's video.endpoint, drop it from everything sent to the
SFU (applyRequestedVideoChannels and the data-channel replay; filtered there
rather than at entry because the app's first request precedes the join
response) and feed own-endpoint sinks from the camera preview, exactly as
CustomImpl does. Before, the engine added a recvonly transceiver whose remote
SSRCs were its own outgoing ones, renegotiated, asked the SFU to loop its
video back, and hung the local tile's sink on a track that could never
receive.
Testbench: the fake platform's makeVideoSource/makeVideoCapturer returned
nullptr, which is why a no-op setVideoCapture went unnoticed; they now build
a FakeVideoCapturer (1280x720 @ 30 fps, tinted by deviceId) so the CLI can
drive engines through the app's VideoCaptureInterface path
(--video-via-capture at join, --video-via-capture-late after the join
response). --request-own-video appends the participant's own endpoint to
every request, the way the app does; validation asks the SFU afterwards
(new GoSfu_QueryRequestedLayer) whether anyone requested itself, and with a
capture feed requires the own-endpoint sink to have received preview frames.
Reference-only capture runs went 0/2 -> 2/2 video pairs, the self-request
count 1 -> 0; CustomImpl passes the same runs, which is what validates the
fake camera and the assertions. Full existing group regression set green.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
1b6cb5c66f |
group: end-to-end encryption in GroupInstanceReferenceImpl
The PeerConnection-based group engine ignored descriptor.e2eEncryptDecrypt, so a
conference routed to it sent plaintext and could not decrypt anything.
The frame layout is now shared rather than reimplemented: FrameTransformer, the
H264/VP8 plaintext-prefix helpers, ValidateEncryptedFrame and four free functions
move out of GroupInstanceCustomImpl.cpp into group/GroupFrameTransformer.{h,cpp},
and the RNNoise capture post-processor into
group/GroupAudioCapturePostProcessor.{h,cpp}. A conference can mix engines, so the
bytes must match CustomImpl exactly; group_frame_transformer_test pins the layout
(the Opus trailer, the one-byte strip when the extension flag is clear, the VP8 and
H264 prefixes, the 3-to-4 byte NAL start-code widening, the four-attempt retry).
The reference engine encrypts both senders and decrypts the per-receiver audio
transformer, each video endpoint's receiver, and the unsignaled-SSRC discovery tap.
Sender ids resolve per frame through a shared ssrc -> userId registry fed by the
requestMediaChannelDescriptions response the engine used to discard, because the
recvonly transceiver is added on a debounce that can precede that response.
Membership in the registry, not a userId value of 0, decides whether a sender is
known. An unresolved SSRC is re-asked, de-duped on the in-flight request as
CustomImpl does; the engine previously asked exactly once ever.
CustomImpl's capture post-processor is installed here too, so the level/speech
trailer carries a real measurement -- with encryption on that trailer is the only
way a CustomImpl peer learns we are speaking. It also makes the engine report the
ssrc-0 self level it never reported and makes setIsNoiseSuppressionEnabled work.
isConference is deliberately not consumed: it is unrelated to encryption and its
one effect in CustomImpl is to drop outgoing video to a single simulcast layer.
Adds --e2e to the CLI's group mode. Verified against a mixed CustomImpl/
ReferenceImpl testbench run; a wrong-userId negative control drives it to 0/2.
(cherry picked from commit
|
||
|
|
1d56755658 |
group: make ReferenceImpl incoming video work in real group calls
Four independent defects, each sufficient on its own to leave a remote
participant's video black, diagnosed from real-call logs and reproduced in
the CLI testbench before being fixed.
1. Renegotiation before the join answer. The app requests video for the
participants it already knows about right after emitJoinPayload. That
renegotiation ran while no transceiver had a mid yet, so CreateOffer
handed every m-line a fresh mid from PeerConnection's monotonic
generator (offer #1 = 0/1/2, re-offer = 3/4/5/6) and SetLocalDescription
rejected it ("order of m-lines in subsequent offer doesn't match").
Nothing retried: later identical requests hit the "endpoint already
known" early-continue. setRequestedVideoChannels now stores the full set
and defers until the join response is applied (onJoined), un-negotiated
endpoints are re-flagged, and renegotiate() refuses to run before join.
2. Constraints dropped on a connecting data channel.
sendReceiverVideoConstraints returned early while the channel was still
opening and nothing replayed it, so the SFU was never asked to forward
the video. onDataChannelStateChanged now resends the stored request.
3. Receive payload types. Group calls never negotiate payload types per
pair: every client sends with CustomImpl's table (VP8 100, VP9 102,
H264 104, RTX at +1). PeerConnection's receive table comes from the
local description, numbered by walking the platform factory's format
list; on iOS that put H265 on 104, so the remote's H264 was fed to the
H265 depacketizer and never decoded. mungeVideoCodecsInOffer pins the
convention on every video m-line of every local offer.
4. Sink lifetime. addIncomingVideoOutput registered the app's sink on the
receiver track by raw pointer and never removed it, although the app
only holds a weak reference because the tile view owning the sink is
recreated on every quality switch: use-after-free in
rtc::VideoBroadcaster::OnFrame on the next decoded frame. Each endpoint
now owns a GRVideoSinkProxy (weak sinks, per-frame lock and prune, the
shape of CustomImpl's VideoSinkImpl), the only object ever registered on
the track, attached once the track exists and detached on removal and
before Close().
Testbench:
- --early-video-request requests video right after emitJoinPayload, the
app's order; the SFU-driven request arrives ~500 ms after the data
channel opens and never exercised that path.
- FakeInterface advertises video formats in the iOS order by default so
PeerConnection's auto-assigned table matches the device's (the builtin
host order put an H264 profile on 104 by coincidence and masked
defect 3); --builtin-codec-order restores the raw order.
- --video-sink-churn replaces every incoming sink five times after frames
flow; segfaulted before defect 4's fix.
- requestVideoFromEndpoints resends the accumulated full request set,
since both engines drop endpoints missing from the list.
Known, not fixed: a host ReferenceImpl SENDER throttles to ~1-3 fps after
the first seconds (41-65 frames in 30 s vs ~750 from CustomImpl senders);
_minOutgoingVideoBitrateKbit is stored but never applied. Documented in
CLAUDE.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
|
3160270a35 |
ReferenceImpl: real audio levels via per-receiver PCM sink
Replace GroupInstanceReferenceImpl's hardcoded 0.1 synthetic audio level (reported for every known SSRC, regardless of whether the participant was actually producing audio) with a real peak-amplitude reading per remote receiver. Implementation: - New GRAudioLevelSink (webrtc::AudioTrackSinkInterface) attached to each remote audio track after onRenegotiationComplete via the new wireRemoteAudioLevelSinks(). Tracks per-window peak amplitude in PCM, normalized to RFC 6464's "0 dBov = full-scale sine wave" reference (peak / (32768/sqrt(2))) so the value is comparable to CustomImpl's pow(10, -dbov/20) reading. - pollAudioLevels() now reads consumeLevel() from each sink and skips entries whose sink has produced no samples since the last poll, eliminating phantom level entries for known-but-silent SSRCs. - buildRemoteAnswer() now includes the remote SSRC on each recvonly audio m-line. Without a signaled SSRC the AudioRtpReceiver calls SetDefaultRawAudioSink and only one of N transceivers' RemoteAudioSource ever receives PCM (the rest stay silent and our sinks never fire). MID is excluded from audio m-lines, so BUNDLE demuxes by SSRC. - Result for a 3000-amplitude sine through Opus: ~0.126, matching CustomImpl's reading. Test infrastructure (CLI): - New --mute-participants <ids> flag (comma-separated participant IDs) keeps a participant muted after join. - ParticipantState gains a per-source-SSRC max-level map. - validateGroupState enforces: no peer may report a muted participant's SSRC at level >= 0.05. The relaxed audio-received check applies only to unmuted participants. This is a pre-requisite for the upcoming SSRC-discovery work (per-receiver level sinks must already exist before onRenegotiationComplete starts wiring them up for tap-discovered SSRCs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
aaa583f1a9 |
feat: add testbench (CLI tool, Go SFU, Dockerfile, docs)
Brings the testbench source into the submodule: - tools/cli/ — C++ CLI test tool (P2P, reflector, group, group-churn modes) - tools/go_sfu/ — Go/Pion SFU library, c-archive linked into tgcalls_cli - Dockerfile — multi-stage Linux container build - CLAUDE.md (top-level), tools/cli/CLAUDE.md, tools/go_sfu/CLAUDE.md — docs Bazel build glue (.bazelrc, MODULE.bazel, third-party BUILD edits, tgcalls_core target) remains in the outer repo since the dependency stack lives there; labels and paths in this repo reference the outer-repo workspace root. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |