Files
tgcalls/tools/cli/group_mode.cpp
isaacandClaude Fable 5.1 f359144db7 group: ReferenceImpl never lets WebRTC create unsignaled receive streams
Two production defects with one cause: a packet the network thread has
already handed to the worker when the worker tears down a receive
stream is received as an unknown SSRC, and WebRTC creates a default
("unsignaled") receive stream for it on a channel whose remote content
never changes again. That stream is never reset and keeps the SSRC's
binding in the Call's RtpDemuxer, so the m-line later negotiated for
the SSRC logs "Sink could not be added" and never receives a packet.

Video (device log 2026-09-18): after the app dropped all video
endpoints and requested them again on the same PeerConnection, one tile
stayed frozen while every decoded frame logged
"VideoReceiveStreamInterface not connected to a VideoSink". The factory
now gets its own FieldTrialsView (GRFieldTrials) that enables
WebRTC-Video-DiscardPacketsWithUnknownSsrc and resolves everything else
through tgcalls' shared FieldTrialBasedConfig; before, the engine
inherited the trial only if a CustomImpl call had run earlier in the
process.

Audio (found in the CLI while reproducing the above): the same race at
the first discovery renegotiation recreated an unsignaled stream on the
mid-0 catch-all right after WebRTC reset it. The participant stayed
audible through that stream, but the level sink on the dedicated
receiver never fired, so peers never saw them as speaking (2 of 6 level
sinks dead in a 3-participant run). The outgoing audio transceiver is
now sendonly (answered recvonly), the GRAudioFrameTransformer catch-all
is gone, discovery goes only through the un-demuxable-packet tap, and
the PeerConnection opts into the new vendored-webrtc seam
RTCConfiguration::disable_payload_type_demuxing. The seam is
load-bearing: without it the lone recvonly audio m-line becomes the
payload-type catch-all and a late unmuter is swallowed there
(--unmute-after scored 0/2). First remote audio now arrives ~280 ms
after the first packet (debounce plus one offer/answer), as with
CustomImpl.

CLI: --video-rerequest drops and re-requests every video endpoint five
times under a 150 ms SFU downlink delay and requires every sink to
receive frames past the count taken just before the re-request
(reproduces the frozen tile with --reference-participants 6: 1/30
stalled on the unfixed engine, both runs). Every group run now also
prints "Audio levels heard: X/Y pairs" (every peer must report a real
level for every unmuted sender) and "Unsignaled audio: N stream(s)
(must be 0)" (engine log scanned for the stream WebRTC creates on its
own; 6 for 3 reference participants before). Full sweep green: audio
3 and 6 reference participants, late unmute (reference and mixed),
mixed e2e with video, reference e2e, custom-only, early video request,
capture feeds, 6-participant re-request, group-churn, bandwidth
step-down scenarios.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

(cherry picked from commit a029775032)

Already in dev/android (via 45fc400/120e384), left as is:
  CLAUDE.md
  tools/cli/CLAUDE.md
  tools/cli/main.cpp

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 18:49:51 +04:00

284 lines
12 KiB
C++

#include "group_mode.h"
#include <cstdlib>
#include "group_participant.h"
#include <algorithm>
#include <chrono>
#include <cstdio>
#include <string>
#include <thread>
#include <unistd.h>
// CGo header
#include "submodules/TgVoipWebrtc/tgcalls/tools/go_sfu/go_sfu.h"
int runGroupMode(int customParticipants, int referenceParticipants, int duration, bool quiet, bool video, const std::string& networkScenario, const std::set<int>& mutedParticipants, bool earlyVideoRequest, bool videoSinkChurn, bool e2e, VideoFeed videoFeed, bool requestOwnVideo, int unmuteAfterSeconds, bool videoRerequest) {
gGroupQuiet = quiet;
gGroupStartTime = std::chrono::steady_clock::now();
int participants = customParticipants + referenceParticipants;
if (participants < 2) {
fprintf(stderr, "Error: need at least 2 participants total\n");
return 1;
}
// The two own the run's timeline and the unmute branch is an `else if` below, so
// combining them would run the scenario, never unmute anyone, find no late
// unmuters to validate, and report SUCCESS without having tested anything.
if (unmuteAfterSeconds > 0 && !networkScenario.empty()) {
fprintf(stderr, "Error: --unmute-after cannot be combined with --network-scenario\n");
return 1;
}
groupLog("Group", "initializing Go SFU...");
int rc = GoSfu_Init();
if (rc != 0) {
fprintf(stderr, "Error: GoSfu_Init failed with %d\n", rc);
return 1;
}
GoInt sfuHandle = GoSfu_Create();
if (sfuHandle <= 0) {
fprintf(stderr, "Error: GoSfu_Create failed\n");
return 1;
}
const char* videoFeedName = videoFeed == VideoFeed::Source ? "source"
: videoFeed == VideoFeed::CaptureAtJoin ? "capture-at-join" : "capture-late";
groupLog("Group", "created SFU handle=%lld, custom=%d, reference=%d, duration=%ds, video=%d, earlyVideoRequest=%d, videoFeed=%s, requestOwnVideo=%d",
(long long)sfuHandle, customParticipants, referenceParticipants, duration, video ? 1 : 0, earlyVideoRequest ? 1 : 0,
videoFeedName, requestOwnVideo ? 1 : 0);
auto threads = tgcalls::StaticThreads::getThreads();
// Create participants
std::vector<std::unique_ptr<ParticipantState>> states;
bool anyFailed = false;
for (int i = 0; i < participants; ++i) {
bool isReference = (i >= customParticipants);
bool muted = mutedParticipants.count(i) > 0;
auto state = createParticipant(i, isReference, sfuHandle, threads, quiet, video, &states, muted, earlyVideoRequest, e2e, videoFeed, requestOwnVideo);
if (!state) {
anyFailed = true;
continue;
}
states.push_back(std::move(state));
}
// Wait for all participants to connect
groupLog("Group", "waiting for connections...");
bool allConnected = false;
auto waitStart = std::chrono::steady_clock::now();
while (std::chrono::steady_clock::now() - waitStart < std::chrono::seconds(15)) {
int connectedCount = 0;
for (const auto& s : states) {
if (s->wasConnected.load()) connectedCount++;
}
if (connectedCount == (int)states.size()) {
allConnected = true;
groupLog("Group", "all %d participants connected", (int)states.size());
break;
}
groupLog("Group", "connected: %d/%d", connectedCount, (int)states.size());
std::this_thread::sleep_for(std::chrono::milliseconds(500));
}
if (!allConnected) {
int connectedCount = 0;
for (const auto& s : states) {
if (s->wasConnected.load()) connectedCount++;
}
groupLog("Group", "connection timeout: %d/%d connected", connectedCount, (int)states.size());
}
// Optionally replace every incoming video sink a few times once frames
// are flowing (mirrors the app recreating tile views on quality changes).
if (video && videoSinkChurn) {
constexpr int kSinkChurnRounds = 5;
groupLog("Group", "video-sink-churn: waiting 3s for frames, then %d replacement rounds", kSinkChurnRounds);
std::this_thread::sleep_for(std::chrono::seconds(3));
for (int round = 0; round < kSinkChurnRounds; ++round) {
groupLog("Group", "video-sink-churn: round %d", round + 1);
churnVideoSinks(states);
std::this_thread::sleep_for(std::chrono::seconds(1));
}
}
// Optionally drop every video request and issue it again on the same
// PeerConnection (the app does this whenever the UI leaves and re-enters
// the video grid, e.g. on backgrounding). Video keeps flowing under a
// downlink delay while the engines renegotiate, because on a real network
// the packets the SFU forwarded before it read the empty
// ReceiverVideoConstraints are still in flight when the receive streams
// are torn down; without the delay the in-process SFU stops the streams
// before the renegotiation lands and the case is never exercised.
//
// The cycle repeats: the hazard is a packet that the network thread has
// already handed to the worker when the worker tears the stream down, and
// at the harness's packet rates a single cycle rarely lands in that
// window. Validation compares against the counts recorded at the last
// drop, so every round must come back for the run to pass.
if (video && videoRerequest) {
constexpr int kEgressDelayMs = 150;
constexpr int kRerequestRounds = 5;
groupLog("Group", "video-rerequest: waiting 3s for frames, then %dms downlink delay and %d drop/re-request rounds", kEgressDelayMs, kRerequestRounds);
std::this_thread::sleep_for(std::chrono::seconds(3));
for (const auto& s : states) {
GoSfu_SetNetworkParams(sfuHandle, s->id, 1, kEgressDelayMs, 0, 0.0, 0);
}
std::this_thread::sleep_for(std::chrono::seconds(1));
for (int round = 0; round < kRerequestRounds; ++round) {
groupLog("Group", "video-rerequest: round %d", round + 1);
dropAllVideoRequests(states);
std::this_thread::sleep_for(std::chrono::seconds(1));
rerequestAllVideo(states);
std::this_thread::sleep_for(std::chrono::seconds(1));
}
}
// Run for the specified duration, optionally with network scenario.
if (!networkScenario.empty() && networkScenario == "step-down-up") {
// Scenario: start uncapped, then step down, step up, uncap.
// Split duration into 4 phases.
int phase = std::max(duration / 4, 2);
groupLog("Group", "network-scenario '%s': phase duration=%ds", networkScenario.c_str(), phase);
// Phase 1: uncapped (should be layer 2 on high BW).
groupLog("Group", "phase 1: uncapped");
std::this_thread::sleep_for(std::chrono::seconds(phase));
// Phase 2: cap to 80 kbps (should force downswitch to layer 0).
groupLog("Group", "phase 2: cap 80kbps");
for (const auto& s : states) {
GoSfu_SetNetworkParams(sfuHandle, s->id, 1, 0, 0, 0.0, 80000);
}
std::this_thread::sleep_for(std::chrono::seconds(phase));
// Phase 3: cap to 200 kbps (should allow upswitch to layer 1).
groupLog("Group", "phase 3: cap 200kbps");
for (const auto& s : states) {
GoSfu_SetNetworkParams(sfuHandle, s->id, 1, 0, 0, 0.0, 200000);
}
std::this_thread::sleep_for(std::chrono::seconds(phase));
// Phase 4: uncap (should allow upswitch to layer 2).
groupLog("Group", "phase 4: uncapped");
for (const auto& s : states) {
GoSfu_SetNetworkParams(sfuHandle, s->id, 1, 0, 0, 0.0, 0);
}
std::this_thread::sleep_for(std::chrono::seconds(phase));
} else if (unmuteAfterSeconds > 0) {
// Mid-call unmute. The muted participants sent nothing while everyone
// joined, so their SSRCs are unknown to every peer until now — and by
// now each peer has already negotiated the audio m-lines for the SSRCs
// it discovered at join. This is the app's "someone turns their
// microphone on after a while" case.
int settle = std::min(unmuteAfterSeconds, duration);
groupLog("Group", "running for %ds, then unmuting %d participant(s)",
settle, (int)mutedParticipants.size());
std::this_thread::sleep_for(std::chrono::seconds(settle));
// Nothing is signalled to the peers: they are told about this participant by
// no one and must notice the new SSRC from the media itself, exactly as they
// must in a real call. The unmute deliberately lands after the peers have
// renegotiated for the SSRCs they saw at join.
for (const auto& s : states) {
if (!s->muted || !s->instance) continue;
groupLog("Group", "unmuting P%d (ssrc=%u)", s->id, s->audioSsrc);
s->instance->setIsMuted(false);
s->muted = false;
s->unmutedLate = true;
}
int remaining = std::max(duration - settle, 5);
groupLog("Group", "running for %d more seconds after the unmute...", remaining);
std::this_thread::sleep_for(std::chrono::seconds(remaining));
} else {
groupLog("Group", "running for %d seconds...", duration);
std::this_thread::sleep_for(std::chrono::seconds(duration));
}
// Ask the SFU, while it is still alive, whether any participant requested
// its OWN video. --request-own-video puts the own endpoint in the list the
// way the app does; the engine has to drop it before it reaches the SFU.
if (video) {
for (const auto& s : states) {
s->sfuSelfRequestedLayer.store((int)GoSfu_QueryRequestedLayer(sfuHandle, (GoInt)s->id, (GoInt)s->id));
}
}
// Ask the SFU how many audio RTP packets each participant actually sent it.
// Validation requires zero for anyone who stayed muted for the whole run.
for (const auto& s : states) {
if (s->audioSsrc != 0) {
s->sfuAudioPacketsReceived.store((int64_t)GoSfu_QuerySsrcPackets(sfuHandle, s->audioSsrc));
}
}
// Stop all participants (using GoSfu_Destroy for bulk teardown)
groupLog("Group", "stopping participants...");
// Stop video sources first
for (auto& s : states) {
if (s->videoSource) {
s->videoSource->Stop();
}
if (s->videoCapture) {
s->videoCapture->setState(tgcalls::VideoState::Inactive);
}
}
// Stop instances
std::atomic<int> stopCount{0};
std::mutex stopMutex;
std::condition_variable stopCv;
for (const auto& s : states) {
if (s->instance) {
int pid_local = s->id;
s->instance->stop([&stopCount, &stopMutex, &stopCv, pid_local]() {
groupLog("Group", "participant %d stopped", pid_local);
stopCount.fetch_add(1);
std::lock_guard<std::mutex> lock(stopMutex);
stopCv.notify_all();
});
}
}
{
std::unique_lock<std::mutex> lock(stopMutex);
stopCv.wait_for(lock, std::chrono::seconds(5), [&] {
return stopCount.load() >= (int)states.size();
});
}
for (auto& s : states) {
s->instance.reset();
s->videoCapture.reset();
}
// Destroy SFU
GoSfu_Destroy(sfuHandle);
GoSfu_Shutdown();
// Validate and print summary
auto result = validateGroupState(states, video);
bool success = printGroupSummary(customParticipants, referenceParticipants, duration, video, result, anyFailed);
// Clean up log files
for (const auto& s : states) {
// TGCALLS_CLI_KEEP_LOGS=1 keeps the engine logs of a passing run
// (see stopParticipant in group_participant.cpp).
if (const char* keep = std::getenv("TGCALLS_CLI_KEEP_LOGS"); !(keep && keep[0] == '1')) {
unlink(s->logPath.c_str());
}
}
fflush(stdout);
fflush(stderr);
_exit(success ? 0 : 1);
}