mirror of
https://github.com/FluidInference/FluidAudio.git
synced 2026-06-11 20:24:36 +00:00
Alex
1dfe1dbd37
Updated descriptions for various models, clarifying features and performance metrics. Enhanced details for TDT, streaming, custom vocabulary, VAD, diarization, and TTS models. ### Why is this change needed? <!-- Explain the motivation for this change. What problem does it solve? -->
7.7 KiB
7.7 KiB
Models
A guide to each CoreML model pipeline in FluidAudio.
ASR Models
Batch Transcription (Near Real-Time)
| Model | Description | Context |
|---|---|---|
| Parakeet TDT v2 | Batch speech-to-text, English only (0.6B params). TDT architecture. | First ASR model added. |
| Parakeet TDT v3 | Batch speech-to-text, 25 European languages (0.6B params). Default ASR model. | Released after v2 to add multilingual support. |
| Parakeet TDT-CTC-110M | Hybrid TDT-CTC batch model (110M params). 3.01% WER on LibriSpeech test-clean. 96.5x RTFx on M2 Mac. Fused preprocessor+encoder for reduced memory footprint. iOS compatible. | Smaller, faster alternative to v3 with competitive accuracy. |
| Parakeet CTC Japanese | Batch speech-to-text, Japanese only (0.6B params). CTC architecture. 6.85% CER on JSUT dataset. 10.1x RTFx on M2 Mac. | First Japanese ASR model. Uses CTC greedy decoder. |
| Parakeet TDT Japanese | Batch speech-to-text, Japanese only (0.6B params). TDT architecture. 6.85% CER on JSUT dataset. 10.8x RTFx on M2 Mac. Hybrid model (CTC preprocessor/encoder + TDT decoder/joint). | Alternative Japanese decoder using TDT for same quality but faster |
| Parakeet CTC Chinese | Batch speech-to-text, Mandarin Chinese (0.6B params). CTC architecture. 8.37% mean CER on THCHS-30 dataset. Int8 encoder (0.55GB) or FP32 (1.1GB). | First Mandarin Chinese ASR model. Uses CTC greedy decoder. |
TDT models process audio in chunks (~15s with overlap) as batch operations.
Streaming Transcription (True Real-Time)
| Model | Description | Context |
|---|---|---|
| Parakeet EOU | Streaming speech-to-text with end-of-utterance detection (120M params). Processes 160ms/320ms frames for true real-time results as the user speaks. | Added after TDT was released & for streaming. Smaller model (120M vs 0.6B). |
| Nemotron Speech Streaming 0.6B | RNNT streaming ASR with 4 chunk size variants (80ms, 160ms, 560ms, 1120ms). English-only (0.6B params). Int8 encoder quantization. Supports ultra-low latency (80ms chunks) to high accuracy (1120ms chunks). | Larger streaming model for better accuracy and quality |
Custom Vocabulary / Keyword Spotting
| Model | Description | Context |
|---|---|---|
| Parakeet CTC 110M | CTC-based encoder for custom keyword spotting. Runs rescoring alongside TDT to boost domain-specific terms (names, jargon). | |
| Parakeet CTC 0.6B | Larger CTC variant (same role as 110M) with better quality |
VAD Models
| Model | Description | Context |
|---|---|---|
| Silero VAD | Voice activity detection; speech vs silence on 256ms windows. Segments audio before ASR or diarization. | Support model that other pipelines build on. Converted at the time being the best model out there |
Diarization Models
| Model | Description | Context |
|---|---|---|
| LS-EEND | Research prototype end-to-end streaming diarization model from Westlake University. Supports both streaming and complete-buffer inference for up to 10 speakers. Uses frame-in, frame-out processing, requiring 900ms of warmup audio and 100ms per update. | Added after Sortformer to support largers speaker counts. |
| Sortformer | NVIDIA's enterprise-grade end-to-end streaming diarization model. Supports both streaming and complete-buffer inference for up to 4 speakers. More stable than LS-EEND, but sometimes misses speech. Processes audio in chunks, requiring 1040ms of warmup audio and 480ms per update for the low latency versions. | Added after Pyannote to support low-latency streaming diarization. |
| Pyannote CoreML Pipeline | Speaker diarization. Segmentation model + WeSpeaker embeddings for clustering. Best offline diarization pipeline, but also support online use | First diarizer model added. Converted from Pyannote with custom made batching mode |
TTS Models
| Model | Description | Context |
|---|---|---|
| Kokoro TTS | Text-to-speech synthesis (82M params), 48 voices, minimal RAM usage on iOS. Generates all frames at once via flow matching over mel spectrograms + Vocos vocoder. Uses CoreML G2P model for phonemization. | First TTS backend added + support custom pronounces |
| PocketTTS | Second TTS backend (~155M params). Autoregressive frame-by-frame generation with dynamic audio chunking. No phoneme stage, works directly on text tokens. | Supports streaming, minimal RAM usage, excellent quality |
Evaluated Models (Not Supported)
Models we converted and tested but are not supported: too large for on-device deployment, limitations or superseded by better approaches.
| Model | Status |
|---|---|
| KittenTTS (FluidAudio#409, HF) | Not supported due to inefficient espeak alternatives. Nano (15M) and Mini (82M) variants. |
| Qwen3-TTS (FluidAudio#290, mobius#20, HF) | Now 1.1GB but too slow. Needs further testing. |
| Qwen3-ForcedAligner-0.6B (FluidAudio#315, mobius#21, HF) | 5-model CoreML pipeline, large footprint. Low upstream adoption (Qwen ASR CoreML model downloads). |