## Summary
Adds **`--multiplex-peer`**, a non-standard relay mode that replaces the
per-allocation peer-side port bind with **one shared IPv4+IPv6 UDP
socket pair per relay thread**. Sessions are demultiplexed by exact peer
IP:port in a per-thread `mp_table`. This lifts the ~16 k allocation cap
that the default 49152-65535 relay port range imposes, and dramatically
reduces kernel-level UDP receive-buffer drops under high pps.
Design and trade-offs: [docs/multiplex-peer.md](docs/multiplex-peer.md).
## What changes
### Server (turnserver)
- **`--multiplex-peer`** (cross-platform) — enable the shared per-thread
relay sockets. Replaces the per-session port bind. Implies sendmmsg
batching on Linux and default-enables `--udp-recvmmsg` (override with
`--udp-recvmmsg=0`). Incompatible with EVEN-PORT — those Allocates are
rejected with 400.
- **`--multiplex-peer-port <port>`** (cross-platform, default 3480) —
base port; thread `i` binds `<base>+2i` (IPv4) and `<base>+2i+1` (IPv6).
A 4-thread server consumes 8 ports.
- **`--udp-gso`** (Linux-only CLI) — UDP-GSO (`UDP_SEGMENT` cmsg) on the
relay send path. Requires `--multiplex-peer` (which is what enables the
sendmmsg batching GSO piggybacks on); passing `--udp-gso` alone is a
silent no-op.
- **CLI surface tightened**: `--udp-recvmmsg`, `--udp-recvmmsg-log`,
`--udp-gso` and their fields are now `#if defined(__linux__)` — absent
from `--help`, rejected with `unrecognized option`, and the code paths
compile out on macOS/Windows.
- **Windows portability**: `SO_REUSEPORT` in `mp_open_socket` wrapped in
`#ifdef` (MSVC's Winsock doesn't define it; REUSEPORT was defensive
anyway because the per-thread port layout is unique by construction).
- **`--sock-buf-size` honoured at startup**: the shared multiplex-peer
relay socket now calls `set_ioa_socket_buf_size` in `mp_open_socket` so
the configured rcvbuf is in effect from the moment the socket exists,
not deferred to the first Allocate.
### turnutils_uclient (loadgen)
- **`--no-even-port`** — force `ep = -1` on Allocate. The default path
randomly attaches EVEN-PORT (with no-R bit) even under `-c`, which
`--multiplex-peer` strictly rejects with 400; this flag makes
alloc-flood runs against multiplex-peer deterministic.
- **Legacy `timer_handler` now wraps the per-tick send batch with
`uclient_send_batch_begin/_end`** — without this, runs with
`--sender-threads 0` (the default for `-m < 4`) silently fell through
every send to plain `send(2)`. strace A/B: 205 k `sendto` → 61 k
`sendmsg` (GSO) + 4 k `sendmmsg` + small `sendto` residual for control.
## Measured impact (3-droplet DigitalOcean, c-4 / 4 vCPU, 8 concurrent
UDP streams, 45 s)
| | baseline | `--udp-recvmmsg` | `--multiplex-peer` | `--multiplex-peer
--udp-gso` |
|---|---:|---:|---:|---:|
| Server NIC rx pps (UDP relay both legs) | 350 k | 334 k | 326 k | 294
k |
| Server `UdpInDatagrams` pps | 279 k | 292 k | 300 k | 294 k |
| **Server `UdpRcvbufErrors` pps** | **71 k** | 42 k | 26 k | **0.3 k
(−99.6 %)** |
| **`turnserver` process CPU** | **387 %** | 205 % | 283 % | **133 %
(−65 %)** |
| Server host idle | 22 % | 49 % | 41 % | **68 %** |
Same loadgen-side packet rate (~2 M pps reported by uclient `send_pps`
after the legacy-path batching fix). Iteration log:
[docs/PerformanceIterationLog.md](docs/PerformanceIterationLog.md).
## Test plan
- [x] `ctest --test-dir build` — 3/3 pass (test_ioaddr, test_stun_msg,
test_http_server) on macOS + Linux.
- [x] `examples/run_tests.sh` — 4 protocols + 4 threaded + load-gen
smoke on Linux; 4 protocols on macOS.
- [x] `examples/run_tests_conf.sh` — same coverage, conf-driven.
- [x] `examples/run_tests_multiplex_peer.sh` — UDP/TCP/TLS/DTLS via
`--multiplex-peer --multiplex-peer-port=35000` on macOS + Linux.
- [x] Flag matrix smoke on macOS: `--multiplex-peer`,
`--multiplex-peer-port=42000`, `--multiplex-peer --udp-gso` (no-op),
`uclient --no-even-port`, `uclient --listener-threads N --sender-threads
M` — all pass; `--udp-recvmmsg` / `--udp-gso` correctly rejected with
`unrecognized option`.
- [x] Flag matrix smoke on Linux (Docker): same + `--udp-recvmmsg`
accepted, `--multiplex-peer` auto-enables `--udp-recvmmsg`,
`--udp-recvmmsg=0` overrides the auto-enable.
- [x] Windows compile fix verified — `SO_REUSEPORT` no longer referenced
unconditionally.
- [x] 3-droplet perf matrix completed; per-hop UDP counters captured.
## Docs updated
- New: [docs/multiplex-peer.md](docs/multiplex-peer.md)
- [README.turnserver](README.turnserver): full entries for
`--multiplex-peer`, `--multiplex-peer-port`, `--udp-gso`; clarified
`--udp-recvmmsg` auto-enable semantics.
- [README.turnutils](README.turnutils): added `--no-even-port`, plus
previously-undocumented `--listener-threads` / `--sender-threads`
loadgen pool flags.
- [examples/etc/turnserver.conf](examples/etc/turnserver.conf):
commented `udp-recvmmsg`, `udp-recvmmsg-log`, `udp-gso`,
`multiplex-peer`, `multiplex-peer-port` keys with one-paragraph
descriptions and pointer to `docs/multiplex-peer.md`.
- Man pages regenerated via `./make-man.sh`.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Extends the existing Linux-only `--udp-recvmmsg` flag from the UDP
listener socket to also cover **connected per-session UDP relay
sockets**, so steady-state client→relay and peer→relay traffic on plain
UDP is read in batches of up to 16 datagrams per `recvmmsg(2)` instead
of one `recvmsg` per packet. DTLS sessions still go through the SSL read
path and are unchanged.
The flag stays **opt-in**: receive-side batching works correctly, but on
the current `m=1` / `m=100` benchmarks throughput is flat to slightly
negative — the bottleneck has moved past receive (see results below).
## What's in the change
- **Shared receive helpers** (`src/apps/relay/ns_ioalib_engine_impl.c`,
`src/apps/relay/ns_ioalib_impl.h`):
- `ioa_parse_udp_recvmsg_cmsg()` — single TTL/TOS/`IP_RECVERR` cmsg
parser used by both `udp_recvfrom()` and the new batch path. Replaces
the duplicated parser previously inlined in `dtls_listener.c` and
`udp_recvfrom()`.
- `ioa_init_recvmmsg_hdr()` — single initializer for
`mmsghdr`/`iovec`/cmsg/source-address fields, also used by the listener.
- New `IOA_UDP_RECVMMSG_MAX_BATCH = 16` constant; both listener and
relay paths now share it.
- **Connected relay batch read** (`socket_udp_read_batch_recvmmsg` in
`ns_ioalib_engine_impl.c`): called from `socket_input_worker` for
non-SSL UDP sockets when `--udp-recvmmsg` is on. Allocates per-message
`stun_buffer_list_elem`s, calls `recvmmsg(MSG_DONTWAIT)`, dispatches
each datagram through the existing `read_cb` path, and falls back
cleanly on `ENOSYS`/`EINVAL`/`EOPNOTSUPP` (auto-disables the flag) and
on `EAGAIN`/short-batch (releases unused buffers).
- **Per-engine scratch state**: the `mmsghdr[16]` / `iovec[16]` / cmsg /
src-addr arrays live on `ioa_engine`, not on every socket — keeps memory
flat at thousands of allocations.
- **TTL/TOS-sized cmsg buffers** in the listener: the listener
previously over-allocated `64 KiB` per slot; it now uses the same
TTL+TOS sizing as the relay path.
- **Opt-in occupancy stats** behind a new `--udp-recvmmsg-log` flag:
every 10 s the relay logs `udp-recvmmsg stats: calls=… packets=…
avg_batch=… wouldblock=… unavailable=… no_buffer=… hist_1=… hist_2=…
hist_3_4=… hist_5_8=… hist_9_16=…`. Counters are always tracked (cheap);
the periodic log is gated by the new flag so default operation is
silent.
- **CLI plumbing**: `--udp-recvmmsg-log` long option in
`mainrelay.c`/`mainrelay.h`, `cli_print_flag` entry in
`turn_admin_server.c`, doc updates in `README.turnserver`.
- **Docs**: `docs/PerformanceIterationLog.md` records the iteration
steps, validation, and two rounds of DigitalOcean A/B numbers.
`CLAUDE.md` load-test instructions updated to mention the new flag and
the `tot_recv_msgs` / `tot_recv_bytes` workaround.
The turnserver man page had drifted from the actual CLI options the
binary accepts. The shipped `man/man1/turnserver.1` was last regenerated
on 05 June 2021, so several options added since then were missing and
one removed option was still documented.
The man page is auto-generated from `README.turnserver` via
`make-man.sh` (txt2man), so the source-of-truth edit is in the README;
the `.1` files are then regenerated.
In `README.turnserver`:
- Add 13 options that exist in `mainrelay.c` long_options[] but were
undocumented: --include-reason-string, --syslog-facility,
--drop-invalid-packets, --drop-invalid-packets-log, --udp-recvmmsg,
--respond-http-unsupported, --prometheus-address, --prometheus-path,
--version, --cpus, --no-cli, --no-rfc5780,
--response-origin-only-with-rfc5780.
- Document --sql-userdb as an alias on the existing --psql-userdb line.
- Remove the stale --ne=[1|2|3] entry (no longer parsed by the binary).
The regenerated `man/man1/turnserver.1` also picks up a backlog of
options that were already in the README but never reached the shipped
page (--software-attribute, --cli, --sock-buf-size, --raw-public-keys,
--stun-backward-compatibility, and the corrected --no-tlsv1_2 wording).
`man/man1/turnadmin.1` and `man/man1/turnutils.1` are regenerated as a
side-effect of `make-man.sh` running over all three READMEs; their
content was similarly stale relative to README.turnadmin /
README.turnutils.
cli interface is ON by default which creates a security risk (even
though requires a password) and recommended to be disabled.
Instead of just recommendation, this PR disables CLI by default and now
requires an explicit flag to enable it
If using old configuration or cli arguments to turnserver - it will log
an error message about `--no-cli` being deprecated while doing nothing
(already disabled). This log line will be removed in the future
# Description
Replace the hardcoded buffer sizes inside coturn to make them
configurable for different use cases (low bitrate use cases can save
memory and high bitrate use case can avoid congestion) - based on #1089
Add this feature in both sides (listener and relay connections).
# Tests
For now it is only the automated CI tests.
Confirmed with debugger that buffer sizes are set according to the
arguments.
Deprecate `--no-stun-backward-compatibility` and set it to true by
default
Add new option `--stun-backward-compatibility`, off by default
Update example/recommended configuration files
This is a breaking change as passing `--no-stun-backward-compatibility`
will be rejected as invalid argument
Invert `--no-rfc5780` option to be true by default
Make it `--rfc5780` to enable it
Update example/recommended configuration files
Passing `--no-rfc5780` will have no effect as this is the default
behavior now
As part of looking at #1588 , I figured that sending `SOFTWARE`
attribute is also part of a problem as it increases messages sent out by
coturn and thus increasing amplification factor. For 4.6.2, the
additional size is 24 bytes (4 bytes attribute header, and 20 bytes for
"Coturn-4.6.2 'Gorst'")
If we are to use an example from #1588, "A 62 byte request will be met
with Coturn’s 401 Unauthorized response which is 150 bytes, a factor of
~2.42." - without SOFTWARE the response will be 126 bytes which reduces
amplification factor to ~2.
As I observed with multiple providers using coturn - some of the are
sending it. Meaning, they do not set `--no-software-attribute` - most
probably due to lack of clarity about this setting.
I believe sending SOFTWARE_ATTRIBUTE should be off by default which is
hinted in the RFC
(https://datatracker.ietf.org/doc/html/rfc8489#section-16.1.2)
Detailed changes:
- Extract setting the attribute into a function to avoid code
duplication
- This option is now not reloadable
- The option is now called `software_attribute` because inverse logic
creates multiple double-not in the code which makes it harder to read.
- `no-software_attribute` is still functional but marked as deprecated
in documentation
Test Plan:
- Run local tests with different cli arguments (new and deprecated) and
confirm SOFTWARE attribute is off by default, and added when arguments
say so
This is not a long term documentation works - this is a temp proposal to
modernize existing documentation
- Create docs folder
- Move existing documentation files into docs
- Split one massive INSTALL files into smaller files (no text editing
was done except small markdown header formatting)
- Rename files to be markdowns for nicer rendering
- Bad md rendering is expected in many places
There are two different API's in OpenSSL for configuring TLS ciphers,
one for TLS 1.2 and below, and another for TLS 1.3. coturn only calls
the TLS 1.2 API when handling the `--cipher-list` configuration option,
which means that it's not possible to use non-default ciphersuites with
TLS 1.3 connections.
This PR calls appropriate OpenSSL API to allow TLS 1.3 ciphersuites to be configured.
This PR fixes some typos and formatting, and regenerates the manual
pages from the README files. These changes were originally included in
#1105, however I've split them out into a separate PR as requested.
openssl allows multiple TLS version support through a single SSL_CTX
object.
This PR replaces 4 per-version SSL_CTX objects with a single object
(DTLS is not yet changed).
SSL context initialization code for openssl with modern API (>=1.1.0)
uses `TLS_server_method` and `SSL_CTX_set_min_proto_version` instead of
enabling specific TLS version. Byproduct of this is TLSv1_3 support when
used with openssl-1.1.1 and above
TLS 1.2 and TLS 1.3 cannot be disabled (as before)
Test plan:
- run_tests.sh script now runs turnserver with SSL certificate (which
enables TLS support)
- run_tests.sh now has one more basic test that uses TLS protocol
Co-authored-by: Pavel Punsky <pavel.punsky@epicgames.com>
While that prometheus exporter was initially enabled by default, it looks like there's been a change of plan, resulting in some inconsistencies in the doc.
The `--no-prometheus` option was replaced by a `--prometheus` one -- according to https://github.com/wolmi/coturn/blob/master/src/apps/relay/mainrelay.c#L540
Nikolayshcxandnmegdanov
czephyr
Molly Miller
Gustavo Garcia
Molly Miller
Benjamin Porter
Lionel Nicolas
Mészáros Mihály
Ruben Barkow-Kuder
Samuel
wolmi
Miquel Ortega
Greg Fodor