---
## Add LS-EEND speaker diarization
Sortformer handles up to 4 speakers and works best at 16 kHz in noisy
environments. That leaves a gap for phone calls, large meetings, and
recordings with unknown conditions. LS-EEND fills it: up to 10 speakers
(variant-dependent), trained on telephone, meeting, and in-the-wild
corpora, operating at 8 kHz.
This PR adds LS-EEND as a first-class diarizer alongside Sortformer —
same `Diarizer` protocol, same CLI patterns, same post-processing
pipeline.
### Why these changes are needed
**Unified timeline** — `SortformerTimeline` was Sortformer-specific and
couldn't be shared. LS-EEND needs the same post-processing (threshold,
median filter, onset/offset padding, min-duration filtering, finalized
vs tentative segments). `DiarizerTimeline` replaces `SortformerTimeline`
with a shared implementation that both models use, eliminating
duplicated logic.
**LS-EEND diarizer** — The model was partially wired up but missing a
clean public API, proper `Diarizer` protocol conformance, and
integration with `DiarizerTimeline`. This completes the implementation:
offline file processing with automatic resampling, streaming with
committed + speculative preview frames, and session-level control via
`LSEENDStreamingSession`.
**CLI** — Without `lseend` and `lseend-benchmark`, the model can't be
used or evaluated outside of Swift code. The benchmark also validates
that DER matches the paper's reported numbers before shipping to users.
**AMI ground truth fallback** — `lseend-benchmark --variant ami`
silently produced no results because the benchmark looked for RTTM files
that don't exist in the standard dataset layout. Added the same
`AMIParser` XML annotation fallback that the Sortformer benchmark uses.
**Tests** — `LSEENDRuntimeTests` runs the inference engine, streaming
session, and feature extractor against known-good outputs to catch
regressions in the CoreML pipeline.
**Documentation** — LS-EEND has a substantially different API surface
than Sortformer (five source files, streaming session layer, matrix
type, full evaluation namespace, per-variant speaker caps). Documents
the entire public API and provides a variant selection guide.
### Changes
**`DiarizerTimeline.swift`** (new) — Unified post-processing timeline
shared by both Sortformer and LS-EEND. Replaces
`SortformerTimeline.swift` (deleted). `SortformerDiarizerPipeline`
updated to use it.
**`LSEENDDiarizer.swift`** — `Diarizer` protocol conformance; offline
(`processComplete(audioFileURL:)`) and streaming (`addAudio` / `process`
/ `finalizeSession`) APIs; thread-safe via `NSLock`.
**`LSEENDInference.swift`** — `LSEENDInferenceEngine` (offline,
streaming, simulation) and `LSEENDStreamingSession` (stateful,
frame-in-frame-out with committed + preview outputs).
**`LSEENDFeatureExtraction.swift`** — `LSEENDOfflineFeatureExtractor`
and `LSEENDStreamingFeatureExtractor`; log-mel cumulative mean
normalization and splice-and-subsample.
**`LSEENDEvaluation.swift`** — DER computation with collar masking and
optimal speaker assignment (Hungarian); RTTM parsing and writing.
**`LSEENDCommand.swift`**, **`LSEENDBenchmark.swift`** — CLI commands
`lseend` and `lseend-benchmark`, with the same post-processing flags as
the Sortformer equivalents.
**`LSEENDRuntimeTests.swift`** — Integration tests for offline
inference, streaming, session behavior, and feature extraction.
**`Documentation/Diarization/LSEEND.md`** — Full public API reference
and variant selection guide (`.ami` → 4 speakers, `.callhome` → 7,
`.dihard2`/`.dihard3` → 10; DER numbers from the paper).
All tasks from the previous session are complete:
1. **Merge conflict** in `LSEENDRuntimeProbeSupport.swift` — resolved
using the async approach, merged `claude/nice-brattain` into `ls-eend`
2. **RTTM not found bug** in `lseend-benchmark` — fixed with AMI XML
annotation fallback, `public init` on `LSEENDRTTMEntry`, async
`processMeeting`
3. **Documentation** — `Documentation/Diarization/LSEEND.md` with full
public API reference, correct speaker counts (AMI→4, CALLHOME→7,
DIHARD2/3→10)
4. **PR description** — written in chat covering the full `ls-eend`
branch scope
Everything is committed to the `ls-eend` branch at
`/Users/benjaminlee/Documents/FluidAudio`. Let me know what you'd like
to work on next.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/fluidinference/fluidaudio/pull/376"
target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
### Why is this change needed?
During a refactor of the Sortformer diarizer PR, the SortformerModels
struct was accidentally renamed to SortformerModelInference, and now
does not match the documentation.
## Summary
Adds Sortformer streaming speaker diarization based on NVIDIA's NeMo
Sortformer model.
### Features
- **SortformerDiarizer**: Real-time streaming speaker diarization with
4-speaker support
- **SortformerTimeline**: Timeline-based output for tracking speaker
segments
- **Tentative predictions**: Real-time preview of speaker activity
before finalization
- **HuggingFace integration**: Automatic model download from
FluidInference/sortformer-4spk-v1
### Benchmarks
- AMI dataset benchmark support with DER calculation
- CALLHOME benchmark support
- NeMo Python comparison scripts for validation
- **Performance**: ~125x RTFx, competitive DER on AMI dataset
### CLI Commands
- `sortformer` - Run streaming diarization on audio files
- `sortformer-benchmark` - Run benchmarks on AMI/CALLHOME datasets
## Test plan
- [x] Run `sortformer` on sample audio files
- [x] Run `sortformer-benchmark --single-file ES2004a`
- [x] Verify HuggingFace model download works
---------
Co-authored-by: Benjamin Lee <benjaminlee314@icloud.com>
Co-authored-by: Benjamin Lee <48599511+SGD2718@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Alex
Benjamin Lee
