Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions Documentation/ASR/NemotronMultilingual.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,60 @@ let detected = await manager.detectedLanguage() // e.g. "fr-FR"
await manager.reset()
```

### Custom vocabulary (hotword biasing)

The Nemotron exports ship no CTC head, so the sliding-window path's CTC
rescorer does not apply here. Instead the manager biases inside the greedy
RNN-T decode: tokens that extend a partially matched vocabulary term get a
flat log-prob bonus at every emission step (decode-time shallow fusion, issue
#841).

```swift
await manager.setCustomVocabulary([
CustomVocabularyTerm(text: "Torvane"),
CustomVocabularyTerm(text: "Quexal", aliases: ["Kwexal"]),
])
// Pass [] to disable. Survives reset(); may be set before loadModels().
```

Semantics and limits:

- **Weight scale.** `weight` here is a *per-token logit bonus* (default 4.5,
the measured recall peak), not the CTC rescoring weight. Values above 6.0
are treated as legacy CTC-scale weights and fall back to the 4.5 default —
the simple text-list loader's blanket `weight: 10.0` would otherwise
over-bias (measured artifacts: word splits like "build today" → "build to
day"). Most terms should omit `weight`; explicit values in (0, 6.0] are
honored.
- **Assets.** Biasing needs a logits-producing step decoder
(`decoder_joint_noencproj` or `decoder_joint`). When a vocabulary is
active the fused-argmax `decoder_joint_argmax` asset is bypassed in favor
of a logits path; if the load-time tier priority skipped `decoder_joint`,
it is loaded lazily from the model directory when the vocabulary is set.
Only if no logits decoder exists at all is the vocabulary rejected, with
an error log rather than silent half-application.
- **Greedy decode.** Once a boosted token wins the argmax it is committed —
there is no beam to undo an over-fire. Keep vocabularies to genuinely
rare terms; a term the model hears as a word it already spells
("Pheynix" → "Phoenix") needs an alias, not more weight.
- **Chunk boundaries are not a barrier.** Unlike the sliding-window CTC
path, the decoder state and the term match state persist across chunks
(only `reset()`/`finish()` clear them), so a term whose audio spans a
boundary keeps its bias. Verified by a boundary-phase sweep: stepping
leading silence through a full 2240 ms chunk period, term recovery holds
at every phase. Multi-word terms are still harder — more greedy steps
must go right — but not because of chunking.
- Terms shorter than 3 letters are skipped (2 for CJK); word-start anchoring
keeps "ran" from matching into "CRAN". CJK terms match unanchored.
- **False-fire profile** (LibriSpeech test-clean spot check, 40 invented
distractor terms): confidently decoded speech is untouched, but an
acoustically uncertain rare-name region can be captured by a distractor
that shares the true audio's opening letters. Keep vocabularies small and
relevant; do not feed speculative or screen-harvested term lists.

`FLUIDAUDIO_BIAS_LOG=1` traces every boosted flip on stderr for weight
tuning and over-fire attribution.

## Benchmark Results

Apple M2, FLEURS test set, int8 encoder, `MLComputeUnits.cpuAndNeuralEngine`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -317,8 +317,10 @@ public struct CustomVocabularyContext: Sendable {
/// Load a vocabulary file, auto-detecting the structured JSON config vs the
/// simple one-term-per-line text format. JSON config files begin with `{`
/// (after optional leading whitespace); anything else is treated as simple
/// text.
static func loadVocabularyFile(at url: URL) throws -> CustomVocabularyContext {
/// text. Detection is by first meaningful byte, not by try-and-fallback,
/// so a malformed JSON config surfaces its parse error instead of being
/// silently reinterpreted as a list of hotwords.
public static func loadVocabularyFile(at url: URL) throws -> CustomVocabularyContext {
let data = try Data(contentsOf: url)
let whitespace: Set<UInt8> = [0x20, 0x09, 0x0a, 0x0d] // space, tab, LF, CR
let firstMeaningfulByte = data.first { !whitespace.contains($0) }
Expand Down
Loading
Loading