fix(asr/nemotron-multilingual): rescue pause-delimited speech spans that decode to all-blank (#838) - #865
Merged
Merged
Conversation
…hat decode to all-blank (#838) The cache-aware encoder + RNN-T decoder carry state across chunks. After certain preceding audio, the greedy decode collapses to blank for an entire short, pause-delimited word and silently drops it (issue #838). Reproduced with the issue's say-clip harness across 19 lead voices: the same 0.39s word was dropped in 33/171 trials, with one lead voice (Karen) dropping it at every gap from 0 to 2000ms. Isolation showed the failure lives in the carried encoder+decoder state jointly: resetting either side alone makes decode worse, resetting both (= fresh session) recovers the word, and the failure persists unchanged at the 1120ms and 2240ms tiers, exonerating chunk geometry. The same span always decodes correctly from fresh state, matching the reporter's standalone-replay finding. Fix: blank-span rescue, mirroring the #861 final-window re-decode on the sliding-window path. The manager tracks speech spans with an 80ms RMS gate (span closes after 160ms of silence, 240ms pre-roll). When a span closes and no lexical token timing falls inside it, the span audio is re-decoded on fresh encoder caches + decoder state (live stream state is saved and restored; forced-prefix language seeding is re-applied) and recovered tokens are appended to the transcript. Punctuation-only emissions do not mask a span: with long pauses the decode often spends the span's frames on the previous sentence's terminal punctuation. A 400ms close-side attribution slack absorbs RNN-T emission latency so a late-emitted word does not get duplicated by the rescue. Results on the repro matrix (19 voices x 9 gaps, 560ms tier): true silent drops 33 -> 7, and all 7 remaining are gap <= 200ms where no VAD-detectable pause exists to split the span (same limitation as the reporter's Silero-based workaround). Remaining non-drop misses are homophone mishears (Gemma -> Jenna), present with and without the fix. Regression A/B (FLUIDAUDIO_DISABLE_BLANK_RESCUE=1 vs default): - FLEURS en_us, 50 files, 560ms: WER 8.2 both, CER 4.4 vs 4.5, RTFx 60.0 vs 59.5 - FLEURS cmn_hans_cn, 20 files, 560ms: WER/CER 17.5/17.5 identical, RTFx 54.6 vs 54.5 The rescue exposes blankRescueCount (nonzero = the live decode silently dropped pause-delimited speech) as a caller-visible signal, per the issue's API ask. Env knobs: FLUIDAUDIO_DISABLE_BLANK_RESCUE opts out, FLUIDAUDIO_RESCUE_RMS_THRESHOLD tunes the silence gate (default 0.0025, 0 disables). Also corrects the stale att_context_size=[56,0] comments: published multilingual artifacts ship [42,13] with channel cache [1,24,42,1024] per their metadata.json.
Supertonic3 Smoke Test ✅
Runtime: 0m26s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 20s • 2026-08-19T08:11:27.833Z |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 48.7s diarization time • Test runtime: 2m 47s • 08/19/2026, 04:15 AM EST |
PocketTTS Smoke Test ✅
Runtime: 0m8s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 134.9s processing • Test runtime: 2m 26s • 08/19/2026, 04:07 AM EST |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m14s • 08/19/2026, 04:04 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 10m13s • 08/19/2026, 04:14 AM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
…cued tokens (#838 review) Addresses the three review findings on PR #865: 1. Ordering (P1): the rescue previously appended recovered tokens to the accumulators, so when a dropped span and later speech shared one 1120/2240ms chunk the transcript read 'later-word rescued-word'. The trial decode is now staged (captured and removed from the accumulators) and committed via mergeRescuedTokens, which inserts at the span's timestamp position with lang-tag-aware id/timing index mapping. Verified with a lead + dropped-word + trailing-speech fixture at all three tiers: 'Gemma thanks everyone', never inverted. 2. Unvalidated commit (P2): the rescue is committed only when the staged output contains lexical content; lang-tag re-emissions are dropped before the check and punctuation-only results leave the transcript untouched. Partial callbacks are suppressed during the trial decode (they would surface unvalidated, out-of-order text) and a single callback fires after a successful commit. Attempted spans are now counted separately: detectedBlankSpanCount increments whenever a live span decodes to all-blank (the caller-visible drop signal even when the rescue also comes back blank), blankRescueCount only on committed lexical recovery. 3. Counter lifecycle (P2): both counters are cleared in resetRescueState(), which runs from resetStates() on reset(), loadModels(), and loadFromShared(). Validation also surfaced a chained-drop bug the review scenario exposed: the rescue decode's own emission latency (plus the zero-pad flush) stamps trailing recovered tokens past the span end, and those timings bled into the NEXT span's attribution window - masking a second consecutive drop (the tail utterance after a rescued word decoded to all-blank at 560ms and was never rescued). Staged timings are now clamped to the span's real extent, after which back-to-back drops rescue independently (det=2/resc=2 on the stress fixture, both words recovered in order). Re-validated: 19-voice x 9-gap sweep unchanged at 16 misses (7 true drops, all gap <= 200ms; rest homophone mishears); FLEURS en_us 50-file A/B remains WER 8.2 / CER 4.5 / RTFx 59.9 (baseline 8.2/4.4/60.0). Tests: mergeRescuedTokens ordering (mid-insert, append-at-end, leading lang-tag offset, empty live), counter clearing across reset().
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #838.
Problem
On the Nemotron multilingual streaming manager, a short word spoken in isolation between pauses can silently decode to all-blank, and whether it does depends on the preceding audio, not the word (see #838 for the original repro and narrowing).
Reproduced with the issue's
say-clip harness, swept across 19 lead voices (the reporter's exact voices didn't fail on this machine —sayoutput differs across macOS versions, and the bug is state-dependent): the same 0.39 s "Gemma" clip was silently dropped in 33 of 171 trials, with one lead voice (Karen) dropping it at every gap from 0 to 2000 ms and a Samantha lead never dropping it — exactly the reported shape.Root cause
[42,13]-lookahead vs 7-frame-chunk theory from the issue thread is exonerated.Fix: blank-span rescue
Same shape as the #861 final-window re-decode on the sliding-window path, applied to pause-delimited spans:
"…afternoon."+ dropped word); punctuation must not mask a swallowed word."Jenna Gemma").The trial decode is staged and validated before anything reaches the transcript: its tokens are captured and removed from the accumulators, partial callbacks are suppressed for its duration (one callback fires after a successful commit), and the result is committed only when it contains lexical content — inserted at the span's timestamp position (lang-tag-aware index mapping), so words already decoded from later audio in the same 1120/2240 ms chunk keep their order. Staged timings are clamped to the span's real extent; without the clamp, the rescue decode's own emission latency stamps trailing tokens past the span end and masks a second consecutive drop (observed on the stress fixture: the utterance right after a rescued word also decoded to all-blank and was silently lost — with the clamp, back-to-back drops rescue independently and in order).
Per the issue's API ask, two counters are public and cleared on
reset():detectedBlankSpanCount(a live span decoded to all-blank — the drop signal, even when the rescue also comes back blank) andblankRescueCount(committed lexical recoveries, always ≤ detected). Env knobs:FLUIDAUDIO_DISABLE_BLANK_RESCUEopts out;FLUIDAUDIO_RESCUE_RMS_THRESHOLDtunes the silence gate (default 0.0025,0disables).Results
Repro matrix (19 voices × 9 gaps, 560 ms tier):
The 7 remaining drops have no VAD-detectable pause to split the span (gap ≤ 200 ms) — the same inherent limitation as the reporter's Silero-based rescue workaround. Remaining non-drop misses are homophone mishears ("Gemma" → "Jenna"), present with and without the fix (that's #841 territory, not a drop).
Regression A/B (
FLUIDAUDIO_DISABLE_BLANK_RESCUE=1vs default), 560 ms tier:Also corrects the stale
att_context_size=[56,0]comments — the published multilingual artifacts ship[42,13]with channel cache[1,24,42,1024]per theirmetadata.json.Not covered (follow-ups)
StreamingNemotronAsrManagerlikely has the same failure class and could take the same rescue; untested here.🤖 Generated with Claude Code