coturn/CLAUDE.md

AGENT.md — Coturn

Coturn is a TURN/STUN server written in C11, implementing RFC 5766, RFC 5389, and related NAT traversal protocols. It supports multiple database backends (SQLite, PostgreSQL, MySQL, Redis, MongoDB), multiple auth mechanisms, and a fuzzing harness for OSS-Fuzz.

Build

# Standard build
mkdir build && cd build
cmake ..
make -j$(nproc)

# Debug build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

# Fuzzer build (requires clang or AppleClang)
CC=clang CXX=clang++ cmake -S . -B build -DFUZZER=ON
cmake --build build -j$(nproc)

Key CMake options:

• -DFUZZER=ON — build OSS-Fuzz targets (requires Clang or AppleClang)
• -DCMAKE_BUILD_TYPE=Debug|Release
• -DWITH_MYSQL=ON/OFF, -DWITH_PGSQL=ON/OFF, -DWITH_MONGO=ON/OFF, -DWITH_REDIS=ON/OFF

Required validation

Every code change must be validated before it is considered complete. Run the unit tests, the local/system tests, and the fuzzing smoke tests. When working from macOS, also validate the Linux build and Docker image tests inside Docker containers.

# Local build + unit tests
cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build --parallel $(nproc)
ctest --test-dir build --output-on-failure

# Local/system tests
cd examples
./run_tests.sh                  # default protocols, both legacy and
                                # threaded uclient (--listener-threads /
                                # --sender-threads). On Linux it also
                                # enables --udp-recvmmsg on the server
                                # and runs a -Y packet load-gen smoke
                                # that checks non-zero send_pps +
                                # recv_pps. GSO/multiplex live in
                                # run_tests_multiplex_peer.sh.
./run_tests_conf.sh             # same protocols, conf-file driven;
                                # mirrors run_tests.sh via config keys.
./run_tests_multiplex_peer.sh   # exercises --multiplex-peer with shared
                                # per-thread relay sockets (UDP/TCP/TLS/
                                # DTLS) on the small port range opened by
                                # --multiplex-peer-port. Auto-enables
                                # --udp-recvmmsg on Linux.
./run_tests_prom.sh             # only when Prometheus support is built
cd ..

# Fuzzing smoke tests (increase runs/time for fuzzing-related changes)
fuzzing/run-local.sh ASan 0 -runs=1
fuzzing/run-local.sh ASan 1 -runs=1

Treat any FAIL line from the example scripts as a test failure even if the script exits with status 0.

On macOS, run the Linux validation in Docker as well. The fuzzing smoke tests build the reusable coturn-fuzz-local image; then run a clean Linux build and system-test pass against a copied checkout so no root-owned build artifacts are left in the working tree:

docker run --rm \
  -v "$PWD:/src:ro" \
  --entrypoint bash \
  coturn-fuzz-local \
  -lc 'apt-get update && apt-get install -y --no-install-recommends git && \
       cp -a /src /tmp/coturn && \
       cd /tmp/coturn && \
       cmake -S . -B build-linux -DBUILD_TESTING=ON && \
       cmake --build build-linux --parallel $(nproc) && \
       ctest --test-dir build-linux --output-on-failure && \
       rm -rf build && ln -s build-linux build && \
       cd examples && ./run_tests.sh && ./run_tests_conf.sh && \
       ./run_tests_multiplex_peer.sh'

Also validate the packaged Docker image:

cd docker/coturn
make docker.image dockerfile=debian tag=codex-local platform=linux/arm64
make test.docker tag=codex-local platform=linux/arm64/v8
cd ../..

Use platform=linux/amd64 on x86_64 hosts. On Apple Silicon, build with platform=linux/arm64 and run the Bats image tests with platform=linux/arm64/v8, which is the spelling expected by docker/coturn/tests/main.bats.

Code style

All C source — including src/, fuzzing/, and tests/ — must be formatted with clang-format-15 using the project's .clang-format. The CI job .github/workflows/clang.yml runs make lint and fails the build on any formatting drift, so any commit containing C/H files must be formatted before it is created.

# Format the entire repo (uses the Makefile target — equivalent to
# `find . \( -iname "*.c" -o -iname "*.h" \) -not -name "ns_turn_khash.h" \
#    | xargs clang-format -i`. ns_turn_khash.h is a vendored 3rd-party
# header and is excluded from both format and lint):
make format

# Verify formatting matches CI (zero output = clean):
make lint

Mandatory pre-commit step for any session that edits C/H files:

find . \( -iname "*.c" -o -iname "*.h" \) -not -name "ns_turn_khash.h" | xargs clang-format -i

Run this before git commit whenever the staged diff touches *.c or *.h, even when only one file was edited. The find form above does not require ./configure to have been run, so it works in worktrees and fresh clones.

Key style rules (LLVM-based):

• Indent: 2 spaces, no tabs
• Column limit: 120
• Pointer alignment: right (int *p)
• Brace style: attach (K&R)
• Zero-initialize stack buffers at declaration: uint8_t buf[N] = {0} or SomeStruct s = {0}

Tests

# Protocol conformance (RFC 5769 test vectors)
cd examples && ./scripts/rfc5769.sh

# Basic TURN relay test (run server first, then client)
cd examples && ./scripts/basic/relay.sh
cd examples && ./scripts/basic/udp_c2c_client.sh

# Full test suite
cd examples && ./run_tests.sh

Load Test on DigitalOcean

Topology — 3 same-region droplets

Run uclient, turnserver, and peer on separate droplets so the loadgen and reflector never compete with the server under test. Last known setup was three Ubuntu 24.04 c-4 (4 vCPU / 8 GB) droplets in nyc1, all in the default-nyc1 VPC (10.116.0.0/24):

Role	Droplet	Private IP
turnserver	coturn-server-perf (id 570437076)	10.116.0.2
uclient (loadgen)	coturn-loadgen-perf (id 570437077)	10.116.0.3
peer (reflector)	coturn-peer-perf (id 571385939)	10.116.0.4

The 2-droplet shape (peer co-located on the loadgen) is fine for smoke runs but burns ~20 % of loadgen CPU on the peer reflector and skews send_pps comparisons across configs.

Never paste DigitalOcean tokens into logs or files. Use a local environment variable such as DIGITALOCEAN_TOKEN.

Flag inventory (relay-side)

• --udp-recvmmsg — Linux recvmmsg() batched receive on the shared fan-in sockets only: the client listener and, under --multiplex-peer, the per-thread shared relay socket (udp_recvmmsg_eligible). Per-session relay sockets stay on the single-recv path. On by default on Linux; disable with --udp-recvmmsg=false (or =0). DTLS session sockets still go through the OpenSSL read path.
• --udp-gso — Linux UDP-GSO (UDP_SEGMENT cmsg) when a sendmmsg batch shares destination and segment size. Requires --multiplex-peer — that mode is what enables the sendmmsg batching GSO piggybacks on; passing --udp-gso alone is a silent no-op.
• --multiplex-peer — replace the per-allocation relay-port bind with a per-thread shared IPv4+IPv6 relay socket pair. Implies sendmmsg batching on Linux; --udp-recvmmsg (on by default) supplies the recvmmsg window the batching piggybacks on. Port layout: thread i binds --multiplex-peer-port + 2*i (IPv4) and +1 (IPv6); a 4-thread server with default base 3480 uses 3480–3487. See docs/multiplex-peer.md for the design.
• --udp-recvmmsg-log — emit udp-recvmmsg stats every 10 s; useful to confirm per-thread batch occupancy.

Flag inventory (uclient)

• -Y packet|invalid|alloc — load-generator mode.
• -c — no rtcp; in clnet_allocate the EVEN-PORT slot is then chosen randomly between 0 and -1 (startuclient.c:440).
• --no-even-port — force ep = -1 unconditionally. Required for alloc-flood runs against --multiplex-peer, which strictly rejects EVEN-PORT with error 400 (ns_ioalib_engine_impl.c:1585).
• -K N / --listener-threads N, --sender-threads N — loadgen-side receive/send pools. Auto: 0 for -m < 4, bumped to 1 listener / 2 sender for -m >= 4. Max 4 each. Use --sender-threads 4 to push the loadgen harder when you're trying to saturate the server.

Source upload + build

# locally
git archive --format=tar HEAD -o /tmp/coturn.tar
for ip in TURN_PUBLIC_IP UCLIENT_PUBLIC_IP PEER_PUBLIC_IP; do
  scp /tmp/coturn.tar root@$ip:/root/coturn.tar &
done; wait

# on each droplet (apt step needed once per droplet; peer needs at least
# turnutils_peer; uclient needs turnutils_uclient; server needs turnserver)
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends build-essential cmake pkg-config \
  libssl-dev libevent-dev libsqlite3-dev libhiredis-dev git iproute2 sysstat

rm -rf /root/coturn && mkdir /root/coturn
tar -xf /root/coturn.tar -C /root/coturn
cmake -S /root/coturn -B /root/coturn/build -DCMAKE_BUILD_TYPE=Release
cmake --build /root/coturn/build \
  --target turnserver turnutils_uclient turnutils_peer -j$(nproc)

Sysctl + ulimit on every droplet

sysctl -w net.core.rmem_max=134217728 net.core.wmem_max=134217728 \
  net.core.netdev_max_backlog=250000
ulimit -n 1048576

Peer fleet (peer droplet, 10.116.0.4)

Bring up 8 turnutils_peer processes on ports 3480..3487 so the loadgen can target multiple distinct peer ports. The Linux peer transparently batches inbound datagrams via recvmmsg() and echoes with GSO-segmented sendmsg() when the batch shares source IP and segment size (see src/apps/peer/udpserver.c), so 8 processes on a 4-vCPU peer droplet stay well below saturation:

pkill -x turnutils_peer 2>/dev/null || true
for p in 3480 3481 3482 3483 3484 3485 3486 3487; do
  nohup /root/coturn/build/bin/turnutils_peer -L 10.116.0.4 -p $p \
    > /root/peer_${p}.log 2>&1 &
done
ss -ulnp | grep -E ':(348[0-7])\b' | wc -l  # expect >= 8

turnserver (server droplet, 10.116.0.2)

Baseline (no fast paths). Add flags from the table above to compare:

pkill -x turnserver 2>/dev/null || true
mkdir -p /root/runs
LABEL=baseline   # or recvmmsg / mpx_peer / mpx_gso
EXTRA=""         # "--udp-recvmmsg" |
                 # "--multiplex-peer --multiplex-peer-port=3480" |
                 # "--multiplex-peer --multiplex-peer-port=3480 --udp-gso"

nohup /root/coturn/build/bin/turnserver \
  --use-auth-secret --static-auth-secret=secret --realm=north.gov \
  --allow-loopback-peers \
  --listening-ip=10.116.0.2 --relay-ip=10.116.0.2 \
  --min-port=49152 --max-port=65535 \
  --no-cli --no-tls --no-dtls \
  --log-file=stdout --simple-log \
  $EXTRA \
  > /root/runs/${LABEL}.turnserver.log 2>&1 &
echo $! > /root/runs/${LABEL}.pid

# Wait for readiness before driving load.
until grep -q "Total auth threads:" /root/runs/${LABEL}.turnserver.log; do
  sleep 0.5
done

Packet-flood matrix workload (uclient droplet, 10.116.0.3)

8 parallel uclient processes, each -m 1 against a distinct peer port (3480..3487). The 1-session-per-(thread, peer-port) shape is what keeps --multiplex-peer's per-thread mp_table collision-free regardless of which relay thread the server's round-robin assigns each session to.

mkdir -p /root/runs
LABEL=baseline
PIDS=()
for pp in 3480 3481 3482 3483 3484 3485 3486 3487; do
  LOG=/root/runs/${LABEL}.par_p${pp}.log
  (timeout -s INT 45s /root/coturn/build/bin/turnutils_uclient \
     -Y packet -m 1 -n 50000000 -l 120 -c --no-even-port \
     --listener-threads 1 --sender-threads 4 \
     -e 10.116.0.4 -r "$pp" -X -g \
     -u user -W secret \
     10.116.0.2 > "$LOG" 2>&1) &
  PIDS+=("$!")
done
for p in "${PIDS[@]}"; do wait "$p"; done

turnutils_uclient logs send_pps=… , recv_pps=… , total_sent=… , total_recv=… periodically (uclient.c:945). Always take the median of in-flight progress lines (drop the first 2-3 warm-up samples and any trailing zero-pps lines), then sum across the 8 streams. Two derived values that matter:

• recv_pps = end-to-end relayed throughput. The canonical comparison metric across configs.
• send_pps − recv_pps = packets dropped somewhere on the round trip. At parity throughput this should match across configs.

3-host CPU instrumentation

Run mpstat on all three droplets (host-wide CPU breakdown) and pidstat on the server droplet (turnserver process slice), pinned to the duration of the loadgen run:

DUR=45

# on turnserver
nohup mpstat 1 $DUR > /root/runs/${LABEL}.server_mpstat.txt 2>&1 & disown
nohup pidstat -h -u -r -p $(cat /root/runs/${LABEL}.pid) 1 $DUR \
  > /root/runs/${LABEL}.server_pidstat.txt 2>&1 & disown

# on uclient
nohup mpstat 1 $DUR > /root/runs/${LABEL}.uclient_mpstat.txt 2>&1 & disown

# on peer
nohup mpstat 1 $DUR > /root/runs/${LABEL}.peer_mpstat.txt 2>&1 & disown

Parse the Average: line — note field offsets differ from the per-sample lines because there's no time column:

awk '$1=="Average:" && $2=="all" {
  printf "user=%5.1f sys=%5.1f softirq=%5.1f idle=%5.1f\n", $3, $5, $8, $NF
}' /root/runs/${LABEL}.server_mpstat.txt

# turnserver process CPU only:
tail -1 /root/runs/${LABEL}.server_pidstat.txt | awk '{
  print "usr="$4" sys="$5" CPU="$8"%"
}'

Reading the matrix (what "good" looks like)

A last clean run (2026-05-16, 3-droplet, 8 streams × 45 s, c-4):

Config	recv pps (8-stream sum)	turnserver CPU	server host idle	server host softirq
baseline	91 k	367 %	24 %	14.1 %
--udp-recvmmsg	99 k	229 % (−38 %)	48 %	2.6 %
--multiplex-peer	97 k	293 % (−20 %)	42 %	13.3 %
--multiplex-peer --udp-gso	96 k	134 % (−63 %)	65 %	0.8 %

The recv-pps ceiling (~95-100 k) is the loadgen-side cap on a c-4 uclient — %sys + %softirq on the uclient host runs at ~70 % per core across all configs, indicating the kernel network stack is saturated. The win shows up as server-side CPU savings, not as more relayed pps. To push the server toward saturation, resize the uclient to c-8 (or larger) first.

Bottleneck identification

For any run, look at the three mpstat Average: lines:

Host idle %	Reading
< 20 % on uclient	uclient is the cap; you're measuring loadgen, not server. Bump uclient size or thread pools.
< 20 % on peer	peer reflector saturated; add more peer processes or move peer to a bigger droplet.
> 50 % on server	server has headroom; you can push it harder if uclient/peer allow.
< 10 % on server	server is the bottleneck — this is what you want when measuring server-side perf changes.

Allocation throughput

# on uclient — alloc-flood; --no-even-port keeps multiplex-peer from
# rejecting every other request with 400.
timeout -s INT 60s /root/coturn/build/bin/turnutils_uclient \
  -Y alloc -m 200 -n 1000 -c --no-even-port \
  -L 10.116.0.3 \
  -u user -W secret \
  10.116.0.2 > /root/runs/${LABEL}.alloc.log 2>&1

# rate:
grep -oE "total_allocations=[0-9]+" /root/runs/${LABEL}.alloc.log \
  | awk -F= '{print $2}' | sort -n | tail -1

Sessions are torn down between iterations, so this measures alloc/s throughput, not concurrent-session cap. Baseline and --multiplex-peer deliver roughly the same rate (~700 alloc/s on c-4); allocation is control-plane and isn't where the fast paths help. The cap-removal property of --multiplex-peer is structural (2 ports per relay thread, total) — cite docs/multiplex-peer.md rather than trying to demo it via a load test.

Invalid-packet flood

# on uclient
timeout -s INT 12s /root/coturn/build/bin/turnutils_uclient \
  -Y invalid -m 1 -l 16 \
  10.116.0.2 > /root/runs/invalid_m1.log 2>&1 || true

-Y invalid bypasses the TURN allocate handshake and sprays malformed datagrams at the listener; useful for the parse/reject hot path. The uclient pacer drops its interval to 100 µs and lifts the per-burst send cap to 4096 in flood modes (uclient.c:899), so even -m 1 produces real load. Restart turnserver after an invalid flood before any allocation test — past runs saw rapid RSS growth.

CPU profile (perf, server droplet)

Build with -DCMAKE_BUILD_TYPE=RelWithDebInfo for readable symbols. Capture once per config you're comparing (fastpath_off, fastpath_on, optionally one-flag-at-a-time):

sysctl -w kernel.perf_event_paranoid=-1 kernel.kptr_restrict=0
pid=$(cat /root/runs/${LABEL}.pid)
perf record -F 99 -g -p "$pid" -o /root/runs/${LABEL}.perf.data -- sleep 30
perf report --stdio -i /root/runs/${LABEL}.perf.data --no-children \
  --sort comm,dso,symbol > /root/runs/${LABEL}_perf.report

Useful summaries

# per-stream median send/recv pps across all 8 streams of a run:
for f in /root/runs/${LABEL}.par_p*.log; do
  grep "send_pps=" "$f" | awk -F'[=, ]' '
    {for (i=1;i<=NF;i++) {if ($i=="send_pps") s=$(i+1)+0;
                          if ($i=="recv_pps") r=$(i+1)+0}
     print s" "r}' | sort -n | awk -v f="$f" '
    {a[NR]=$0} END {print f": median="a[int(NR/2)]}'
done

# turnserver process RSS over time:
ps -o pid,rss,vsz,pcpu,pmem,comm -p $(cat /root/runs/${LABEL}.pid)

# server NIC counters (drops show up as nonzero rx_dropped on eth1):
ssh root@TURN_PUBLIC_IP 'cat /proc/net/dev | column -t | head -5'

Unit tests (Unity, opt-in via BUILD_TESTING=ON)

Unity is fetched on demand via CMake FetchContent; nothing is vendored. Tests live under tests/ and link against the existing turnclient static library.

# CMake direct
cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build -j --target check     # builds tests, runs ctest
cmake --build build -j --target test_ioaddr  # build a single binary
ctest --test-dir build --output-on-failure   # run already-built tests

# Legacy Makefile bridge (after ./configure; requires cmake on PATH)
make unit-tests   # bootstraps build/unit-tests/, builds + runs Unity tests

Adding a new test: drop tests/test_<name>.c and append coturn_add_test(test_<name>) in tests/CMakeLists.txt. The check target picks it up automatically.

See docs/Testing.md for database setup and extended test scenarios.

Source layout

src/
  client/          # TURN client library (C)
  client++/        # TURN client library (C++)
  server/          # Core TURN/STUN server logic
  apps/
    relay/         # turnserver main process, listeners, netengine
    uclient/       # CLI test client
include/turn/      # Public headers
fuzzing/           # OSS-Fuzz targets and seed corpora
examples/          # Test scripts and sample configs
turndb/            # Database schema and setup scripts
docs/              # Protocol notes and configuration docs

Common patterns

• Port types: use uint16_t for port fields and parameters (not int); port 0 means OS-assigned ephemeral
• Buffer initialization: zero-initialize stack buffers at declaration (= {0}), not just before first use
• HMAC output buffers: declare as uint8_t buf[MAXSHASIZE] = {0} — the buffer is written into the message before HMAC runs, so uninitialized bytes would be briefly present in the packet
• Uninitialized structs: use = {0} for stack-allocated address structs (e.g., ioa_addr)
• Counter overflow in turn_ports.c: _turnports uses uint32_t low/high counters; comparisons must be overflow-safe (use subtraction, not >=)
• Port bounds checks: use <= USHRT_MAX (not < USHRT_MAX) when validating that an int holds a valid port — port 65535 is valid
• Error handling: check return values of all OpenSSL/libevent calls; use ERR_clear_error() before HMAC operations
• Logging: use TURN_LOG_FUNC macros, not fprintf/perror