feat(asr/nemotron): decode-time custom vocabulary biasing (no CTC head required) - #866
Merged
Conversation
…d required) Implements issue #841: token-level shallow fusion inside the Nemotron multilingual streaming greedy RNN-T decode, based on the engine contributed by the Alma team, with the changes requested in review: - Scalability: fresh-start candidates (offset 0, decode-state independent) are precomputed once at init; per-step continuation matching walks tail suffixes through a trie over term piece-forms, so per-emission cost is bounded by tail length and live partial matches, not vocabulary size. - Weight safety: CustomVocabularyTerm.weight predates this engine and is a CTC-rescoring scale (the simple text-list loader assigns 10.0). Applied raw as a per-token logit bonus it over-biases, so overrides are clamped to maxBoost=6.0 with a one-time warning; default boost is 4.5 (the measured recall peak, matching the tuned CTC rescorer cbw). - Fused-argmax (B2) handling is explicit, not silent: with a vocabulary active, decode prefers a logits-producing step decoder over decoder_joint_argmax; if B2 is the only step decoder, the vocabulary is rejected with an error log (all-or-nothing, no half-applied biasing). - Every argmax site is biased and traced: legacy per-frame loop (B3/B1/ bare-pair), speculative scan, and all drain branches report boosted flips via FLUIDAUDIO_BIAS_LOG=1. - Match state resets on reset()/finish(); terms survive like the selected language and may be set before models load. CLI: nemotron-multilingual-transcribe gains --custom-vocab (JSON config or one-term-per-line text). Validated on real speech (LibriSpeech test-clean, cached 2240ms B1 assets): 7127-75947-0033 baseline misspells all three rare names (Levallier / Tenerchalte / Bragalone); with the vocabulary, Tonnay Charente and Bragelonne substantially recover, with blank-overtake flips visible in the trace. Neutrality: a 40-term invented-distractor vocabulary leaves a neutral utterance byte-identical with zero flips; on the rare-name clip, distractors sharing the true audio's opening letters can contest the uncertain region — documented as the false-fire profile, with guidance to keep vocabularies small and relevant. Unit tests cover word-start anchoring (#702 over-fire mode), CJK unanchored 2-cluster terms, Devanagari conjunct rejoin in scalar space, multi-offset overlap, alias/weight clamp semantics, special-token immunity, reset lifecycle, biased selection incl. blank overtake, and a 2000-term scaling smoke.
PocketTTS Smoke Test ✅
Runtime: 0m8s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Supertonic3 Smoke Test ✅
Runtime: 0m28s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
…m vocabulary benchmark Runs each earnings22-kws chunk twice with the same manager (empty vocabulary vs the chunk's dictionary terms) and scores both with the presence-based TP/FP/FN scheme ctc-earnings-benchmark uses, so the Nemotron decode-time biasing path and the CTC rescoring path are comparable on the same rig. First 50 chunks, default boost 4.5, multilingual 2240ms B1 assets: baseline WER 21.58% vocab recall 29.8% (TP=28 FN=66) FP=0 RTFx 83.4x biased WER 20.07% vocab recall 48.9% (TP=46 FN=48) FP=0 RTFx 83.2x Recall +19.1 points with WER improving 1.5 points and no measurable decode overhead from the matcher. Two per-file regressions observed, both the greedy over-boost artifact class: an already-correct term perturbed by its own boost (Latam->Latan, Andres->andress).
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 24s • 2026-08-19T08:26:33.248Z |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 151.3s processing • Test runtime: 2m 39s • 08/19/2026, 04:19 AM EST |
… biasing The 'decoded as two independent halves' caveat was inherited from the sliding-window CTC path, where windows really are re-decoded independently. In the Nemotron streaming manager both the decoder state and the bias match tail persist across chunk boundaries (only reset()/finish() clear them), so cross-boundary terms keep their bias. Verified empirically: stepping leading silence through a full 2240ms chunk period (0-1960ms, 280ms steps) on LibriSpeech 7127-75947-0033 sweeps the boundary through the 'Tonnay Charente' audio; biased recovery holds at every phase (Tonnay in 6/8, full Tonnay Charente at two phases) while the unbiased baseline never recovers it. Also records the full 772-chunk earnings22-kws paired benchmark: baseline WER 19.77% recall 39.3% (TP=494 FN=762) FP=1 RTFx 88.2x biased WER 18.68% recall 59.7% (TP=750 FN=506) FP=3 RTFx 88.0x
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 11m46s • 08/19/2026, 04:39 AM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m56s • 08/19/2026, 04:32 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 60.7s diarization time • Test runtime: 2m 50s • 08/19/2026, 04:37 AM EST |
1. B2 no longer blocks the logits fallback it advertises. The load path's tier priority skips decoder_joint_noencproj/decoder_joint when decoder_joint_argmax exists, so a bundle shipping B2 alongside a logits decoder still had biasing rejected (or silently falling to the slow bare pair). rebuildVocabularyBias now lazily loads decoder_joint from the remembered model directory before rejecting. B1 specifically: it takes the plain encoder step every decode path has, whereas B3 also needs an encoder that emits encoder_proj — loading a model the decode loop can't feed would silently fall back to unbiased B2, the exact failure this avoids. setCustomVocabulary/rebuildVocabularyBias become async for the lazy load. 2. Legacy weights above maxBoost fall back to the 4.5 default instead of clamping to the aggressive 6.0 ceiling. The simple text-list loader blankets every term with CTC-scale weight 10.0, so plain text files were silently pinned to the hottest allowed boost with no way to request the measured default. Explicit weights in (0, 6.0] are still honored. Verified live: a text-list vocabulary now logs the fallback warning and decodes identically to the 4.5 JSON run. 3. Malformed JSON vocabulary files now surface their parse error instead of being reinterpreted as one-hotword-per-line text. CustomVocabularyContext.loadVocabularyFile (first-meaningful-byte format detection) is now public and the CLI uses it; the try/fallback in nemotron-multilingual-transcribe is gone. Verified live: truncated JSON aborts the run with the decode error.
✅ Nemotron Multilingual Benchmark — FLEURSFLEURS
Logs (tail) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #841.
Implements decode-time shallow-fusion hotword biasing for
StreamingNemotronMultilingualAsrManager, based on the engine contributed inline in #841 by @vinod-memfold (Alma) — thank you for the detailed writeup, measurements, and field-testing addenda. This upstreams that design with the changes requested in the issue review.What's here
NemotronVocabularyBias(new, beside the manager): matches vocabulary terms as lowercased-NFC piece text in Unicode-scalar space, so whichever segmentation the decoder drifts toward stays boostable. Word-start anchoring for space-delimited scripts (the CTC custom-vocabulary rescorer over-fires on short keyword terms since 0.14.8 (PR #634) #702 over-fire guard), unanchored 2-cluster terms for CJK, every viable match offset tracked.CustomVocabularyTerm.weightpredates this engine and is a CTC-rescoring scale (the simple text-list loader assigns10.0). Overrides are clamped tomaxBoost = 6.0with a one-time warning; the default boost is 4.5, the measured recall peak and the same cbw the tuned CTC rescorer settled on. Most terms should omitweight.decoder_joint_noencproj/decoder_joint/ bare pair) overdecoder_joint_argmax; when B2 is the only step decoder, the vocabulary is rejected with an error log. All-or-nothing — no silently half-applied biasing.FLUIDAUDIO_BIAS_LOG=1reports each boosted flip with both logits.reset()/finish(); terms survive like the selected language and may be set before models load.nemotron-multilingual-transcribe --custom-vocab <file>(JSON config or one-term-per-line text).Documentation/ASR/NemotronMultilingual.md.Real-speech validation (review ask)
LibriSpeech test-clean, multilingual 2240ms B1 assets, debug build:
7127-75947-0033(three rare French names) baseline: "How is it Levallier? said Mademoiselle de Tenerchalte that the Vicomte de Bragalone spoke of you as Louise".{Valliere, Tonnay Charente, Bragelonne}: Tonnay Charente and Bragelonne substantially recover (Tonnay Chaunt/Bragelonat 6.0), driven by blank-overtake flips visible in the trace. "Levallier" is the documented alias-needed case — the model already spells it as a word it knows.1089-134686-0000) byte-identical with zero flips, at both 4.5 and 6.0.Tenerchalte→Tone Chaunt) at both 4.5 and 6.0. This narrows Custom vocabulary for Nemotron streaming: decode-time RNN-T biasing (no CTC head required) #841's "0 false fires" claim (measured on 8+6 synthetic clips): confidently decoded speech is safe; contested OOV-name regions are not. Documented in the manager docs with guidance to keep vocabularies small and relevant — and it is why screen-harvested/speculative term lists (per the issue's own screen-feedback trap note) should be filtered before reaching this API.Tests
20-case XCTest suite against a toy piece table (no CoreML): word-start anchoring, CJK unanchored terms, Devanagari conjunct rejoin in scalar space, multi-offset overlap through repeated leading words, alias/weight-clamp semantics, special-token immunity, reset lifecycle, biased selection including blank overtake, and a 2000-term scaling smoke.
Notes for reviewers
CustomVocabularyTermstays where it is to avoid churning open PRs (feat(asr/unified): vocabulary boosting on the unified Parakeet path #862) — moving it to shared ASR vocabulary code is a follow-up.weight: 10.0should map to the 4.5 default instead of clamping to 6.0 is the one open design call; clamping keeps monotonic semantics but is the aggressive end of the measured range.Benchmark: earnings22-kws paired baseline-vs-biased (full 772 chunks, 3.2 h)
New
nemotron-vocab-benchmarkCLI command runs every chunk twice with the same manager (empty vocabulary vs the chunk's dictionary terms), scored with the same presence-based TP/FP/FN scheme asctc-earnings-benchmark. Default boost 4.5, multilingual 2240 ms B1 assets:The 1120 ms tier doubles the chunk-boundary density yet gains the same recall (+19.0 vs +20.4 points) at the same WER improvement — corroborating the boundary-phase sweep: chunking is not the limiting factor for biasing.
Recall +20.4 points with WER improving 1.1 points; 2 net false fires over 3.2 hours; matcher overhead unmeasurable (RTFx flat, validating the trie + precomputed fresh-start design). Per-file regressions observed are the greedy over-boost class — an already-correct term perturbed by its own boost (
Latam→Latan,Andres→andress).Cross-chunk terms: supported (issue's caveat corrected)
#841 carried over the CTC path's "decoded as two independent halves" chunk-boundary caveat. It does not apply here: the decoder state and the bias match tail both persist across chunk boundaries (only
reset()/finish()clear them). Verified by a boundary-phase sweep — stepping leading silence through a full 2240 ms chunk period (280 ms steps) on LibriSpeech7127-75947-0033sweeps the boundary through the "Tonnay Charente" audio, and biased recovery holds at every phase (full "Tonnay Charente" at two phases) while the unbiased baseline never recovers it. Docs updated accordingly.Review fixes (c44696f)
Three findings from a local review pass, all addressed:
decoder_joint_noencproj/decoder_jointwhendecoder_joint_argmaxexists, so a bundle shipping B2 plus a logits decoder still had biasing rejected or falling back to the slow bare pair.rebuildVocabularyBiasnow lazily loadsdecoder_jointfrom the remembered model directory before rejecting (B1 specifically, because it takes the plainencoderstep every decode path has; B3 would also need anencoder_proj-emitting encoder and could silently strand the bias on B2).maxBoost(the text-list loader's blanket CTC-scale 10.0) now fall back to the measured 4.5 default with a warning, instead of clamping to the hottest allowed value; explicit weights in (0, 6.0] are honored. This also settles the open design call flagged above.CustomVocabularyContext.loadVocabularyFile(first-meaningful-byte format detection) is now public and the CLI uses it.Fixes 2 and 3 verified live (text-list run now logs the fallback and decodes identically to the 4.5 JSON run; truncated JSON aborts with the decode error). Fix 1 is compile-verified — no B2 asset exists locally to e2e it.
🤖 Generated with Claude Code