1 Commits

Author	SHA1	Message	Date
Alex	2ea0727541	ASR: fix Parakeet TDT v3 emitting Cyrillic for short Latin-script utterances (#512) (#515) ... Fixes #512. ## TL;DR Parakeet TDT v3 transcribed short Polish utterances like "Wpisz Google kropka com" as Cyrillic (`Впиш Гугл к ком.`) because the joint decoder's top-1 pick drifts to Cyrillic tokens under low acoustic confidence. This PR adds an **opt-in** script filter: when a caller passes `language: .polish` (or any other language with a declared script), the decoder rejects top-1 if it's the wrong script and walks top-K to the highest-probability candidate matching the expected script. - **Opt-in**: `language:` defaults to `nil` — zero behavior change for existing callers. - **No acoustic-model changes** — this is purely a decoder-side post-processing step over the joint logits. - **Requires `JointDecisionv3.mlmodelc`** (exposes top-K outputs). Auto-downloaded from HuggingFace alongside the other v3 files; falls back to standard argmax when absent. ## Empirical validation — reporter's own audio Samples pulled via `gdown --folder <link-from-issue-#512-comment>` from @tajchert's Drive folder. **`JointDecisionv3.mlmodelc` is loaded in both columns** — this isolates the Swift filter as the mechanism, not a model swap. | sample | ground truth | `language: nil` (current) | `language: .polish` (this PR) | |---|---|---|---| | pl | Wpisz Google kropka com | **Впиш Гугл к ком.** | Wpis Google.com. | | pl2 | Wpisz Google kropka com | **Впиш Гугл крокаком.** | Wpish Google, Com. | | pl3 | Wpisz Google kropka com | **Впишь куглькрабком.** | VP Kugl.com. | | pl4 | Wpisz Google kropka com | **Впиш гугл к ком.** | Wpish gugl c. | | pl5 | Wpisz Google kropka com | **Впиш гугл кракаком.** | Wpish Google Croca kom. | | pl6 | Wpisz Google kropka com | **Впиш, гугл крокаком.** | Wpish, Google, Com. | | pl_complex | Cały spichlarz jest ze spiżu | Cały spichlarz jest ze spiżu. | Cały spichlarz jest ze spiżu. | **6/6 short samples flip Cyrillic → Latin.** `pl_complex` was never broken (long context → high joint confidence → no drift) and is unchanged. ## Scope & limitations (important — please don't overclaim) **This PR fixes the *script* the tokens are drawn from. It does NOT fix per-word acoustic accuracy.** | | `language: nil` | `language: .polish` | |---|---|---| | Script correct (Latin, not Cyrillic) | ✗ | ✓ (6/6) | | Word spelling matches ground truth | ✗ | ✗ (still 6/7 wrong on short) | The residual errors — `Wpisz` → `Wpish`/`Wpis`, `kropka` → `Croca` / dropped — are **Parakeet TDT v3 acoustic weaknesses on short Polish commands**. No amount of output post-processing can turn `Wpish` into `Wpisz`; that needs better acoustic modeling, a Polish LM rescorer, or more training data. Out of scope here. What users actually get by merging: - Output is visually Polish (Latin script), not pseudo-Russian — works with locale-aware post-processing, spell-check, and UI rendering - Locale-strict WER evaluators no longer penalize Cyrillic-vs-Latin substitution - Opt-in; zero risk for callers who don't pass `language:` What users do **not** get: - Higher word accuracy on short Polish/Slavic Latin utterances - Support for languages outside the `Language` enum (Greek, Maltese, Hungarian, Turkish, Baltic — their characters fit the Latin Unicode ranges but aren't exposed; easy follow-up) - A meaningful FLEURS WER delta — see [Documentation/fleurs-script-filtering-comparison.md](./Documentation/fleurs-script-filtering-comparison.md); full sentences aren't in the failure regime ## Implementation ### New - `Sources/FluidAudio/Shared/ScriptDetection.swift` (new, +112) - `public enum Language` — 13 Latin (en, es, fr, de, it, pt, ro, pl, cs, sk, sl, hr, bs) + 5 Cyrillic (ru, uk, be, bg, sr) - `public enum Script { case latin, cyrillic }` - `matches(_:script:)` over Unicode ranges: ASCII (0x20–0x7F), Latin-1 (0xA0–0xFF), Latin Extended-A (0x100–0x17F), **Latin Extended-B (0x180–0x24F — Romanian ș/ț)**, **Latin Extended Additional (0x1E00–0x1EFF — Vietnamese)**, Cyrillic (0x400–0x4FF). Strips SentencePiece boundary marker U+2581 before checking. - `filterTopK(topKIds:topKLogits:vocabulary:preferredScript:) -> (tokenId, probability)?` — returns the highest-probability top-K candidate matching the target script; probability via **softmax over the top-K subset** with the max-logit stability trick; guarded against top-K array length mismatch. ### Changed - `TdtJointDecision` — optional `topKIds` / `topKLogits` fields (populated by JointDecisionv3 only) - `TdtDecoderV3` — script filter runs **only when top-1 is already wrong script**; both decode sites feed `filtered.probability` (a real [0,1]) into `TdtDurationMapping.clampProbability`, not raw logits - `AsrManager.transcribe(...)` — `language: Language? = nil` plumbed through all three overloads: `[Float]`, `URL`, `AVAudioPCMBuffer` - `AsrModels` + `ModelNames` — `requiredModelsV3` set includes `JointDecisionv3.mlmodelc` so the download utility fetches it on fresh installs and also backfills it for existing users on next `.v3` load - CLI — `fluidaudiocli transcribe <file> --language {en|pl|cs|sk|sl|hr|bs|ro|es|fr|de|it|pt|ru|uk|be|bg|sr}` ### How to try it ```bash swift run -c release fluidaudiocli transcribe sample.wav --language pl ``` ## Model dependency `JointDecisionv3.mlmodelc` must be present in `FluidInference/parakeet-tdt-0.6b-v3-coreml` on HuggingFace. It exposes `top_k_ids` / `top_k_logits` outputs (K=64 in our export) alongside the standard argmax. When absent, `AsrModels` falls back to `JointDecision.mlmodelc` and the script filter becomes a no-op — backward compatible. **Cache-upgrade verified**: removed `JointDecisionv3.mlmodelc` from a populated cache, re-ran `--language pl`; the file was auto-fetched and Polish output was Latin. Existing users pick up the fix on next `.v3` load without manual intervention. ## Review notes / risky bits - **Softmax over top-K subset, not the full vocab** — probabilities won't exactly match a true full-softmax, but K=64 captures ~all the mass when the model is anywhere near confident. If you prefer, we can expose the raw top-K logits to callers and let them compute confidence however they want. - **Top-1 escape hatch**: filter is only triggered when top-1 fails `matches(_, script:)`. When top-1 is already correct, nothing is changed — so we can't regress the common case. - **Length-mismatch guard** in `filterTopK` uses `min(topKIds.count, topKLogits.count)`. If CoreML output arrays ever diverge, we iterate the common prefix instead of crashing. - **Latin Extended-B (0x0180–0x024F)** was added specifically so Romanian ș/ț aren't rejected as non-Latin. Latin Extended Additional (0x1E00–0x1EFF) was added for free — helps Vietnamese should anyone want it later. ## Tests - `ScriptDetectionTests` — **37 tests**: Unicode range coverage (Latin-1 / Extended-A / Extended-B / Extended Additional / Cyrillic), SentencePiece boundary-marker stripping, `filterTopK` happy path, length-mismatch guard, probability-range invariant, Czech/Slovak/Slovenian/Croatian/Romanian token coverage, cross-script rejection - Build clean; `swift format lint` clean on all touched files - A/B end-to-end run against reporter's actual Polish audio (table above) ## Checklist - [x] Builds clean (`swift build`, `swift build -c release`) - [x] `swift format lint` clean on touched files - [x] `ScriptDetectionTests` 37/37 pass - [x] A/B reproduction on #512 reporter's audio - [x] Cache-upgrade path verified (JointDecisionv3 auto-fetched on existing caches) - [x] CLI accepts all 18 language codes end-to-end - [ ] CI green ## Follow-ups (not blocking) - Expose more Latin languages in the enum (Hungarian, Turkish, Baltic, Maltese) — all character ranges already supported, just need enum cases - Add `Script.greek` for `el_gr` (separate Unicode range) - Short-utterance benchmark dataset (FLEURS is the wrong tool — it's all long sentences where drift doesn't happen) - Optional: publish a Polish LM rescorer to address the underlying acoustic-accuracy issue the script filter cannot fix ---------	2026-04-23 17:43:09 -04:00