mirror of
https://github.com/FluidInference/FluidAudio.git
synced 2026-06-11 20:24:36 +00:00
Julian Pscheid
ab29cead3c
### Why is this change needed? The Nemotron streaming RNNT decode loop already knows which encoder frame each token is emitted at, but the streaming managers only return the decoded text. So there is no way for a caller to line transcript words up with anything time-based (diarization, word-level highlighting, subtitle timing) without running a separate forced alignment. This change surfaces the timing that is already computed. At each token-emission site, next to the existing `accumulatedTokenIds.append(...)`, it records a `TokenTiming` whose `startTime` is the token's absolute encoder-frame index times `ASRConstants.secondsPerEncoderFrame` (80 ms per frame). Callers get the timings from a new `finishWithTokenTimings() -> (text, timings)`, with a `getTokenTimings()` accessor for salvaging a partially decoded session that cannot safely `finish()`. The existing `process()` / `finish()` String API does not change. Both managers are covered: - `StreamingNemotronAsrManager` (English): one emission site in the per-frame decode loop. - `StreamingNemotronMultilingualAsrManager`: three sites, since it has the legacy per-frame loop plus the default-on speculative decoder (first-hit and multi-emission drain). Two extra details here. Language-tag tokens, which `decode()` already strips from the transcript, are kept out of the timing stream so it stays 1:1 with the visible tokens. And a new `NemotronMultilingualTokenizer.rawToken(for:)` passthrough exposes the raw `▁`-marked piece, because the existing `tokenizerPiece` helper strips that marker and callers need it to find word starts. A VAD-skipped chunk still advances the frame base by its nominal frame count so the timeline does not drift. Grouping tokens by the SentencePiece `▁` marker yields word-level `(text, startTime, endTime)` spans. In our app we use that to assign each transcript word to the diarizer segment it overlaps, which fixed boundary-word speaker misattribution on imported audio. A few notes: - Purely additive. No existing API or behavior changes. - `swift format lint` is clean on the changed files (the two pre-existing `K` constant warnings are untouched upstream code). - Granularity is per encoder frame (80 ms), which matches the greedy RNNT decode. There is no per-token duration here like the TDT path has.