mirror of
https://github.com/coturn/coturn.git
synced 2026-06-11 09:44:33 +00:00
## Summary Adds **`--multiplex-peer`**, a non-standard relay mode that replaces the per-allocation peer-side port bind with **one shared IPv4+IPv6 UDP socket pair per relay thread**. Sessions are demultiplexed by exact peer IP:port in a per-thread `mp_table`. This lifts the ~16 k allocation cap that the default 49152-65535 relay port range imposes, and dramatically reduces kernel-level UDP receive-buffer drops under high pps. Design and trade-offs: [docs/multiplex-peer.md](docs/multiplex-peer.md). ## What changes ### Server (turnserver) - **`--multiplex-peer`** (cross-platform) — enable the shared per-thread relay sockets. Replaces the per-session port bind. Implies sendmmsg batching on Linux and default-enables `--udp-recvmmsg` (override with `--udp-recvmmsg=0`). Incompatible with EVEN-PORT — those Allocates are rejected with 400. - **`--multiplex-peer-port <port>`** (cross-platform, default 3480) — base port; thread `i` binds `<base>+2i` (IPv4) and `<base>+2i+1` (IPv6). A 4-thread server consumes 8 ports. - **`--udp-gso`** (Linux-only CLI) — UDP-GSO (`UDP_SEGMENT` cmsg) on the relay send path. Requires `--multiplex-peer` (which is what enables the sendmmsg batching GSO piggybacks on); passing `--udp-gso` alone is a silent no-op. - **CLI surface tightened**: `--udp-recvmmsg`, `--udp-recvmmsg-log`, `--udp-gso` and their fields are now `#if defined(__linux__)` — absent from `--help`, rejected with `unrecognized option`, and the code paths compile out on macOS/Windows. - **Windows portability**: `SO_REUSEPORT` in `mp_open_socket` wrapped in `#ifdef` (MSVC's Winsock doesn't define it; REUSEPORT was defensive anyway because the per-thread port layout is unique by construction). - **`--sock-buf-size` honoured at startup**: the shared multiplex-peer relay socket now calls `set_ioa_socket_buf_size` in `mp_open_socket` so the configured rcvbuf is in effect from the moment the socket exists, not deferred to the first Allocate. ### turnutils_uclient (loadgen) - **`--no-even-port`** — force `ep = -1` on Allocate. The default path randomly attaches EVEN-PORT (with no-R bit) even under `-c`, which `--multiplex-peer` strictly rejects with 400; this flag makes alloc-flood runs against multiplex-peer deterministic. - **Legacy `timer_handler` now wraps the per-tick send batch with `uclient_send_batch_begin/_end`** — without this, runs with `--sender-threads 0` (the default for `-m < 4`) silently fell through every send to plain `send(2)`. strace A/B: 205 k `sendto` → 61 k `sendmsg` (GSO) + 4 k `sendmmsg` + small `sendto` residual for control. ## Measured impact (3-droplet DigitalOcean, c-4 / 4 vCPU, 8 concurrent UDP streams, 45 s) | | baseline | `--udp-recvmmsg` | `--multiplex-peer` | `--multiplex-peer --udp-gso` | |---|---:|---:|---:|---:| | Server NIC rx pps (UDP relay both legs) | 350 k | 334 k | 326 k | 294 k | | Server `UdpInDatagrams` pps | 279 k | 292 k | 300 k | 294 k | | **Server `UdpRcvbufErrors` pps** | **71 k** | 42 k | 26 k | **0.3 k (−99.6 %)** | | **`turnserver` process CPU** | **387 %** | 205 % | 283 % | **133 % (−65 %)** | | Server host idle | 22 % | 49 % | 41 % | **68 %** | Same loadgen-side packet rate (~2 M pps reported by uclient `send_pps` after the legacy-path batching fix). Iteration log: [docs/PerformanceIterationLog.md](docs/PerformanceIterationLog.md). ## Test plan - [x] `ctest --test-dir build` — 3/3 pass (test_ioaddr, test_stun_msg, test_http_server) on macOS + Linux. - [x] `examples/run_tests.sh` — 4 protocols + 4 threaded + load-gen smoke on Linux; 4 protocols on macOS. - [x] `examples/run_tests_conf.sh` — same coverage, conf-driven. - [x] `examples/run_tests_multiplex_peer.sh` — UDP/TCP/TLS/DTLS via `--multiplex-peer --multiplex-peer-port=35000` on macOS + Linux. - [x] Flag matrix smoke on macOS: `--multiplex-peer`, `--multiplex-peer-port=42000`, `--multiplex-peer --udp-gso` (no-op), `uclient --no-even-port`, `uclient --listener-threads N --sender-threads M` — all pass; `--udp-recvmmsg` / `--udp-gso` correctly rejected with `unrecognized option`. - [x] Flag matrix smoke on Linux (Docker): same + `--udp-recvmmsg` accepted, `--multiplex-peer` auto-enables `--udp-recvmmsg`, `--udp-recvmmsg=0` overrides the auto-enable. - [x] Windows compile fix verified — `SO_REUSEPORT` no longer referenced unconditionally. - [x] 3-droplet perf matrix completed; per-hop UDP counters captured. ## Docs updated - New: [docs/multiplex-peer.md](docs/multiplex-peer.md) - [README.turnserver](README.turnserver): full entries for `--multiplex-peer`, `--multiplex-peer-port`, `--udp-gso`; clarified `--udp-recvmmsg` auto-enable semantics. - [README.turnutils](README.turnutils): added `--no-even-port`, plus previously-undocumented `--listener-threads` / `--sender-threads` loadgen pool flags. - [examples/etc/turnserver.conf](examples/etc/turnserver.conf): commented `udp-recvmmsg`, `udp-recvmmsg-log`, `udp-gso`, `multiplex-peer`, `multiplex-peer-port` keys with one-paragraph descriptions and pointer to `docs/multiplex-peer.md`. - Man pages regenerated via `./make-man.sh`. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
204 lines
7.7 KiB
Bash
Executable File
204 lines
7.7 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
TURNSERVER_LOG="/tmp/run_tests_conf.$$.turnserver.log"
|
|
PEER_LOG="/tmp/run_tests_conf.$$.peer.log"
|
|
UCLIENT_LOG="/tmp/run_tests_conf.$$.uclient.log"
|
|
LOADGEN_LOG="/tmp/run_tests_conf.$$.loadgen.log"
|
|
|
|
cleanup() {
|
|
kill "$turnserver_pid" "$peer_pid" 2>/dev/null
|
|
wait "$turnserver_pid" "$peer_pid" 2>/dev/null
|
|
rm -f "$TURNSERVER_LOG" "$PEER_LOG" "$UCLIENT_LOG" "$LOADGEN_LOG"
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
# Detect cmake build and adjust path
|
|
BINDIR="../bin"
|
|
if [ ! -f $BINDIR/turnserver ]; then
|
|
BINDIR="../build/bin"
|
|
fi
|
|
|
|
IS_DARWIN=0
|
|
if [ "$(uname -s)" = "Darwin" ]; then
|
|
IS_DARWIN=1
|
|
fi
|
|
|
|
echo "Creating $BINDIR/turnserver.conf file"
|
|
echo "use-auth-secret" > $BINDIR/turnserver.conf
|
|
echo "static-auth-secret=secret" >> $BINDIR/turnserver.conf
|
|
echo "realm=north.gov" >> $BINDIR/turnserver.conf
|
|
echo "allow-loopback-peers" >> $BINDIR/turnserver.conf
|
|
if [ $IS_DARWIN -eq 0 ]; then
|
|
echo "sock-buf-size=1048576" >> $BINDIR/turnserver.conf
|
|
fi
|
|
echo "cert=../examples/ca/turn_server_cert.pem" >> $BINDIR/turnserver.conf
|
|
echo "pkey=../examples/ca/turn_server_pkey.pem" >> $BINDIR/turnserver.conf
|
|
if [ $IS_DARWIN -eq 0 ]; then
|
|
# Force log output to stdout (which we redirect to $TURNSERVER_LOG below).
|
|
# Without this, turnserver writes to its platform-default location
|
|
# (syslog or /var/log/turn_*.log) and our log file stays empty, which
|
|
# breaks wait_for_turnserver's "Total relay threads:" probe and leaves
|
|
# the FAIL diagnostics useless. simple-log keeps the format compact.
|
|
echo "log-file=stdout" >> $BINDIR/turnserver.conf
|
|
echo "simple-log" >> $BINDIR/turnserver.conf
|
|
# Server-side fast paths: enable on Linux so the conf-driven test
|
|
# cycle also exercises the recvmmsg drain path. The udp-gso path
|
|
# lives behind multiplex-peer (that mode is what enables sendmmsg
|
|
# batching, which GSO piggybacks on); exercising GSO would flip the
|
|
# server into multiplex-peer mode and change the relay-port model
|
|
# the protocol tests below assume. run_tests_multiplex_peer.sh
|
|
# covers the multiplex path.
|
|
if [ "$(uname -s)" = "Linux" ]; then
|
|
echo "udp-recvmmsg" >> $BINDIR/turnserver.conf
|
|
fi
|
|
fi
|
|
|
|
echo 'Running turnserver'
|
|
if [ $IS_DARWIN -eq 1 ]; then
|
|
$BINDIR/turnserver -c $BINDIR/turnserver.conf > /dev/null &
|
|
else
|
|
$BINDIR/turnserver -c $BINDIR/turnserver.conf > "$TURNSERVER_LOG" 2>&1 &
|
|
fi
|
|
turnserver_pid="$!"
|
|
echo 'Running peer client'
|
|
if [ $IS_DARWIN -eq 1 ]; then
|
|
$BINDIR/turnutils_peer -L 127.0.0.1 -L ::1 -L 0.0.0.0 > /dev/null &
|
|
else
|
|
$BINDIR/turnutils_peer -L 127.0.0.1 -L ::1 -L 0.0.0.0 > "$PEER_LOG" 2>&1 &
|
|
fi
|
|
peer_pid="$!"
|
|
|
|
# Wait for OUR turnserver instance to finish init -- see run_tests.sh
|
|
# for the rationale. We poll the per-invocation log (uniquely named via
|
|
# $$) for "Total auth threads:" so a stale turnserver from a prior
|
|
# script invocation can't false-positive us.
|
|
wait_for_turnserver() {
|
|
local i
|
|
for i in $(seq 1 40); do
|
|
if grep -q "Total auth threads:" "$TURNSERVER_LOG" 2>/dev/null; then
|
|
return 0
|
|
fi
|
|
if ! kill -0 "$turnserver_pid" 2>/dev/null; then
|
|
echo "FATAL: turnserver (pid $turnserver_pid) exited before init completed"
|
|
echo "--- turnserver log ---"
|
|
cat "$TURNSERVER_LOG" 2>/dev/null || echo "(log file missing)"
|
|
return 1
|
|
fi
|
|
if ! kill -0 "$peer_pid" 2>/dev/null; then
|
|
echo "FATAL: turnutils_peer (pid $peer_pid) exited before tests started"
|
|
echo "--- peer log ---"
|
|
cat "$PEER_LOG" 2>/dev/null || echo "(log file missing)"
|
|
return 1
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
echo "FATAL: turnserver never reached 'Total auth threads:' init line within 20s"
|
|
echo "--- turnserver log (last 30 lines) ---"
|
|
tail -30 "$TURNSERVER_LOG" 2>/dev/null || echo "(log file missing)"
|
|
return 1
|
|
}
|
|
if [ $IS_DARWIN -eq 1 ]; then
|
|
sleep 5
|
|
else
|
|
wait_for_turnserver || exit 1
|
|
# No-barrier builds can log readiness before all worker event loops have
|
|
# had a scheduling turn. Keep the old startup cushion after the active
|
|
# per-process readiness check.
|
|
sleep 2
|
|
fi
|
|
|
|
# See run_tests.sh for rationale — same shape, mirrored here so the
|
|
# conf-driven test produces the same actionable failure output.
|
|
diagnose_failure() {
|
|
local label="$1"
|
|
echo "=== Diagnostics for failed test: $label ==="
|
|
echo "--- uclient: tot_send/tot_recv progress (last 6 lines) ---"
|
|
grep "start_mclient: tot_send_msgs" "$UCLIENT_LOG" 2>/dev/null | tail -6
|
|
echo "--- uclient: errors / warnings ---"
|
|
grep -iE "error|warning|fail|broken|drop" "$UCLIENT_LOG" 2>/dev/null | tail -10
|
|
echo "--- turnserver: socket / relay / allocation errors (last 40 lines) ---"
|
|
grep -iE "508|capacity|socket|bind|available ports|relay|allocation" "$TURNSERVER_LOG" 2>/dev/null | tail -40
|
|
echo "--- turnserver log (last 20 lines) ---"
|
|
if [ -s "$TURNSERVER_LOG" ]; then
|
|
tail -20 "$TURNSERVER_LOG"
|
|
elif [ -e "$TURNSERVER_LOG" ]; then
|
|
echo "(turnserver log exists but is empty — likely never produced output)"
|
|
else
|
|
echo "(turnserver log missing — redirect path: $TURNSERVER_LOG)"
|
|
fi
|
|
echo "--- peer log (last 10 lines) ---"
|
|
if [ -s "$PEER_LOG" ]; then
|
|
tail -10 "$PEER_LOG"
|
|
elif [ -e "$PEER_LOG" ]; then
|
|
echo "(peer log exists but is empty)"
|
|
else
|
|
echo "(peer log missing — redirect path: $PEER_LOG)"
|
|
fi
|
|
echo "==="
|
|
}
|
|
|
|
# Same factoring as run_tests.sh: function-per-test, run each protocol
|
|
# with the legacy single-threaded uclient and again with the listener +
|
|
# sender thread pools engaged. Total bytes stay at 1000 either way at
|
|
# -m 1 -n 5.
|
|
run_uclient() {
|
|
local label="$1"
|
|
shift
|
|
echo "Running $label"
|
|
"$BINDIR/turnutils_uclient" "$@" -e 127.0.0.1 -X -g -u user -W secret 127.0.0.1 > "$UCLIENT_LOG" 2>&1
|
|
if grep -q "start_mclient: tot_send_bytes ~ 1000, tot_recv_bytes ~ 1000" "$UCLIENT_LOG"; then
|
|
echo OK
|
|
else
|
|
echo FAIL
|
|
diagnose_failure "$label"
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
# Legacy single-threaded uclient.
|
|
run_uclient "turn client TCP" -t
|
|
run_uclient "turn client TLS" -t -S
|
|
run_uclient "turn client UDP"
|
|
run_uclient "turn client DTLS" -S
|
|
|
|
if [ $IS_DARWIN -eq 1 ]; then
|
|
sleep 2
|
|
exit 0
|
|
fi
|
|
|
|
# Listener + sender thread pools engaged at minimum non-zero size.
|
|
run_uclient "turn client TCP (threaded)" -t --listener-threads 1 --sender-threads 1
|
|
run_uclient "turn client TLS (threaded)" -t -S --listener-threads 1 --sender-threads 1
|
|
run_uclient "turn client UDP (threaded)" --listener-threads 1 --sender-threads 1
|
|
run_uclient "turn client DTLS (threaded)" -S --listener-threads 1 --sender-threads 1
|
|
|
|
# Linux-only load-gen smoke (see comment in run_tests.sh for rationale).
|
|
if [ "$(uname -s)" = "Linux" ]; then
|
|
echo "Running turn client UDP load-gen smoke (-Y packet, threaded)"
|
|
timeout -s INT 6s "$BINDIR/turnutils_uclient" \
|
|
-Y packet -m 4 -l 100 -c -e 127.0.0.1 -g \
|
|
--listener-threads 1 --sender-threads 2 \
|
|
-u user -W secret 127.0.0.1 > "$LOADGEN_LOG" 2>&1
|
|
rc=$?
|
|
if [ $rc -ne 0 ] && [ $rc -ne 124 ] && [ $rc -ne 130 ]; then
|
|
echo "FAIL: uclient exited with $rc"
|
|
echo "--- load-gen log ---"
|
|
cat "$LOADGEN_LOG"
|
|
echo "--- turnserver log (last 20 lines) ---"
|
|
tail -20 "$TURNSERVER_LOG"
|
|
exit 1
|
|
fi
|
|
if grep -qE "send_pps=[1-9][0-9]*\.[0-9]+, recv_pps=[1-9][0-9]*\.[0-9]+" "$LOADGEN_LOG"; then
|
|
echo OK
|
|
else
|
|
echo "FAIL: no non-zero send_pps/recv_pps line in load-gen output"
|
|
echo "--- load-gen log (last 20 lines) ---"
|
|
tail -20 "$LOADGEN_LOG"
|
|
echo "--- turnserver log (last 20 lines) ---"
|
|
tail -20 "$TURNSERVER_LOG"
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
sleep 2
|