feat(pocket-tts): per-language mimi encoder conversion for voice cloning (FluidAudio #793) - #84
Open
Alex-Wengg wants to merge 1 commit into
Open
feat(pocket-tts): per-language mimi encoder conversion for voice cloning (FluidAudio #793)#84Alex-Wengg wants to merge 1 commit into
Alex-Wengg wants to merge 1 commit into
Conversation
…ing (FluidAudio #793)
Every pocket-tts language pack ships its own mimi codec weights (all 87
mimi state-dict keys differ across languages, including the 43
encoder-side keys), so the shared English-traced mimi_encoderv2 produced
wrong-codec latents for non-English live cloning — the root cause of the
residual flaky/garbled output that the FluidAudio #797 host-side
reprojection could not fix.
convert_mimi_encoder_lang.py traces one encoder per language (that pack's
mimi encoder + its flow_lm.speaker_proj_weight baked in) with the same
fixed I/O contract as the deployed root encoder (audio [1,1,240000] ->
conditioning [1,125,1024]) and checks CoreML parity against the reference
_encode_audio on a real clip before accepting the artifact (all 9
non-English packs: meanAbsDiff <= 1.7% relative, fp16-level).
Trace-safety shims (pocket_tts 2.1.0): beartype.claw must be neutralized
before first import; apply_rope and the stateless attention forward are
replaced with numerically identical static-shape versions because their
shape arithmetic records int-tensor ops coremltools rejects. The parity
reference is computed before installing the shims so it cannot mask them.
Published as v2.1/<lang>/mimi_encoderv3.{mlpackage,mlmodelc} on
FluidInference/pocket-tts-coreml. Consumed by FluidAudio PR #871.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to FluidInference/FluidAudio#871.
Every pocket-tts language pack ships its own mimi codec weights (all 87 mimi state-dict keys differ across languages, 43 of them encoder-side), so the shared English-traced
mimi_encoderv2produced wrong-codec latents for non-English live cloning — the root cause of the residual flaky/garbled non-English cloning that the FluidAudio #797 host-side reprojection could not fix.convert_mimi_encoder_lang.py:flow_lm.speaker_proj_weightbaked in (output lands directly in that pack's conditioning space — no host reprojection)audio [1,1,240000]→conditioning [1,125,1024], fp16_encode_audioon a real clip before accepting each artifact (all 9 non-English packs ≤1.7% relative, fp16-level)beartype.clawneutralized pre-import;apply_rope+ stateless attention forward swapped for numerically identical static-shape versions (their shape arithmetic records int-tensor ops coremltools rejects). The parity reference is computed before the shims are installed so it can't mask them.Artifacts published as
v2.1/<lang>/mimi_encoderv3.{mlpackage,mlmodelc}on FluidInference/pocket-tts-coreml.End-to-end result (reporter's 5.15 s reference, spanish_24l, 16 seeds): 7/16 intelligible with 4 noise runaways → 15/16, 0 runaways — parity with the PyTorch reference.
🤖 Generated with Claude Code