Upgrade llama.cpp to b10319: migrate TTS from OuteTTS to Qwen3-TTS - #381
Merged
Conversation
Stops one tag before the Qwen3-TTS architecture replacement (b10270, upstream #26254) that guts tools/tts/tts.cpp and removes the OuteTTS pipeline this project's generate-tts-upstream.cmake extracts from.
…n3-TTS
Upstream #26254 deleted the OuteTTS pipeline entirely (tools/tts/tts.cpp
shrank ~1450->205 lines) and replaced it with a single backbone+mmproj
Qwen3-TTS design driven by the new mtmd_helper::gen_audio streaming API.
mtmd_gen_audio_type has exactly one non-NONE value, so Qwen3-TTS is the
only model family the new pipeline supports and there is no OuteTTS path
left to extract from upstream at this tag.
Per explicit direction (no backward compatibility needed; a clean API
that forwards to llama.cpp is preferred over vendor-freezing or dropping
the feature), TextToSpeech/tts_engine.{h,cpp} are fully reworked onto
mtmd_helper::gen_audio:
- tts_engine now loads a backbone (text) model + an mmproj GGUF (speaker
encoder + code predictor + code2wav decoder) and drives set_input() ->
step_prompt() -> step_gen() -> get_output(), requesting raw PCM so the
project's own tested WAV writer (tts_wav.hpp) stays in the loop.
- The OuteTTS build-time extraction (generate-tts-upstream.cmake,
tts_upstream.h) is deleted; mtmd was already linked into jllama for
vision/audio-input, so no new CMake wiring was needed.
- TextToSpeech's public API changes: constructor takes
(modelPath, mmprojPath, ...) instead of (ttcPath, vocoderPath, ...);
synthesize() gains optional speaker-reference/language voice-cloning
parameters.
- test_tts_wav.cpp's WAV-writer tests are unchanged (our own code); the
dead OuteTtsCodecFilter tests are removed (filter_outetts_codec_tokens
no longer exists). TtsIntegrationTest's WAV-format assertions are
unchanged; only its setup/constructor and property names
(tts.model/tts.mmproj) change.
- CI has no Qwen3-TTS model download yet: huggingface.co is network-
blocked in this environment, so exact GGUF/mmproj filenames could not
be verified. The stale OuteTTS/WavTokenizer models.csv rows are
removed (not replaced); TtsIntegrationTest self-skips in CI like
AudioInputIntegrationTest until a session with HF access adds a
verified Qwen3-TTS row.
Full local verification: fresh configure + full cmake --build (libjllama
+ jllama_test link cleanly against the real b10270 mtmd_helper::gen_audio
API) + ctest 482/482 passing (485 minus the 3 removed dead tests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Only priority-8 touch: common_sampler_init drops its n_ctx param and llama_sampler_init_dry drops n_ctx_train (the -1=context-size sentinel for penalty_last_n/dry_penalty_last_n is removed). No project-source change needed: tts_engine.cpp's common_sampler_init call already used the 2-argument form. Configure-only verification (patches reapply clean); full build deferred to the next checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Drops patch 0009 (subprocess.h old-glibc guard): upstream merged the exact fix it submitted (sheredom/subprocess.h#104) via #26606, so the patch is now redundant and its context no longer matches after a neighboring Windows argv-quoting rewrite (confirmed fail-loud "does not apply cleanly" before removing it). Also fixes a real break carried over unbuilt from the b10270->b10275 step (that step was verified configure-only, so a full build never caught it until this checkpoint): eval_llama_cmpl_schema (tools/server/server-schema.h, included directly by jllama.cpp) dropped its n_ctx_slot parameter at b10275, along with the -1=ctx-size sentinel for penalty_last_n/dry_penalty_last_n (hard lower limit moved -1 -> 0). jllama.cpp's populate_completion_task drops its now-unused n_ctx_slot parameter; test_server.cpp's parse_params helper drops its n_ctx parameter and the two *_ExpandsToNCtxSlot tests become *_MinusOne_Throws (a request-supplied -1 is now out-of-range instead of expanding). Added tools/server/server-schema.h to CLAUDE.md's priority-ordered API-compat review list so a directly-included tools/server/*.h header isn't missed again by future bump steps. Full local verification (checkpoint): fresh configure + full cmake --build (libjllama + jllama_test link cleanly) + ctest 482/482 passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Completes the b10275->b10280 step (the previous commit only carried the patch-0009 removal due to a git-add pathspec ordering mistake): pin files (CMakeLists.txt GIT_TAG, README.md, CLAUDE.md, LlamaCppVersion.java) plus the eval_llama_cmpl_schema/n_ctx_slot fix and its docs/history/llama-cpp-breaking-changes.md rows, as described in the prior commit's message. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Purely additive speculative-decoding metrics on server_task_result_metrics/ server_metrics (tools/server/server-task.h, server-context.cpp) plus new Prometheus counters; no signature or field removed. Configure-only verification (patches reapply clean); full build deferred to the next checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Internal hardening fix to server-tools.cpp's file_glob_search directory walk (#26626); server-tools.h unchanged, no project-source impact. Configure-only verification (patches reapply clean); full build deferred to the next checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Internal-only range: CUDA/Vulkan backend tweaks (ggml-cuda.cu, ggml-vulkan.cpp), llama-model-loader.cpp, and Python conversion scripts. No header this project includes directly was touched. Configure-only verification (patches reapply clean); a supplementary full-build checkpoint is running after 3 consecutive configure-only steps. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Follow-up to the b10290->b10295 commit: the supplementary full-build checkpoint (kicked off after 3 consecutive configure-only steps) finished clean. Records the confirmed result in docs/history/llama-cpp-breaking-changes.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
mtmd.h gains a purely-additive enum sentinel (appended last, no renumbering) and two new save/load functions unrelated to this project's mtmd_helper::gen_audio TTS usage; server-models.cpp's change is an internal router-proxy header-forwarding fix. Configure-only verification (patches reapply clean); full build deferred to the next checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Internal-only range: CUDA/SYCL backend tweaks. No header this project includes directly was touched. Configure-only verification (patches reapply clean); full build deferred to the next checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
New SYCL kernels (DSv4 host-conjugate, lightning-indexer) internal to the SYCL backend; diff size dominated by the auto-generated docs/ops/SYCL.csv op-support matrix. No header this project includes directly was touched. Full-build checkpoint (penultimate step before b10310) running; result to follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Follow-up to the b10303->b10308 commit: the penultimate-step full-build checkpoint finished clean. Records the confirmed result. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Smallest, safest step of the entire b10217->b10310 walk: an ARM CPU-feature-detection fix (ggml-cpu/arch/arm/cpu-feats.cpp, #25554) and a WebUI-only TypeScript change (#26709). No header this project includes directly was touched. The mandatory full-build+ctest verification for this final target is running in the background; result to follow in a confirmation commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Closes the b10217->b10310 version-bump walk. Records the confirmed full-build+ctest result for the final target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Adds the two ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF files to
.github/models.csv (Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf backbone,
0.96 GiB; mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf mmproj, 0.42 GiB —
the smallest quants published; there is no Q4_K_M mmproj). Repo
contents (6 files, unauthenticated, no gating) and both resolve URLs
were verified externally against the live HF API and a range-request
GGUF magic-byte check, since huggingface.co is network-blocked in this
sandbox.
Wires TTS_MODEL_NAME/TTS_MMPROJ_NAME env vars and the matching
-Dnet.ladenthin.llama.tts.{model,mmproj} properties into every Java
test job in publish.yml (Linux, all three macOS jobs, both Windows
jobs), the same way the vision/nomic models are wired — so
TtsIntegrationTest is now REQUIRED (validate-models.{sh,bat} hard-fails
on a missing file) and runs on every platform instead of self-skipping.
No change needed to validate-models.{sh,bat} or the download-models job
(both are already manifest-driven off models.csv) or the Android
emulator jobs' free-disk steps (they already delete-by-exclusion,
keeping only the draft model, so a new manifest entry needs no extra
accounting there).
Updates CLAUDE.md/README.md's CI model policy + property tables and
docs/history/llama-cpp-breaking-changes.md's b10269-b10270 row to
record the resolution, since the initial TTS rework had left this
wiring as a documented follow-up (no HF access in that session).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
tools/server/server-models.{h,cpp} gains a router-mode LRU scheduling
queue and an optional should_stop predicate on ensure_model_ready
(default nullptr, source-compatible); patches/0008
(LLAMA_SERVER_WORKER_CMD override) touches the same function this range
touches but still applies cleanly, verified present in the checked-out
source. tools/mtmd/mtmd-helper-gen.cpp's internal overlay-application
rewrite in mtmd_helper::gen_audio doesn't change the public API
tts_engine.cpp calls.
Full local verification (extra-safety checkpoint given the diff's size
and overlap with patch 0008's target): fresh configure + full
cmake --build + ctest 482/482 passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Comment-only fix in server-schema.cpp plus cosmetic ggml-version sync; rest of the range is WebUI-only. No project-source change. Configure-only verification (patches reapply clean); full build deferred to the b10319 final target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
Closes the b10310->b10319 follow-up walk (3 steps). This last step's diff is internal to the clip vision-encoder preprocessing path (tools/mtmd/clip-model.h,clip.cpp,mtmd-image.cpp), unrelated to the Qwen3-TTS pipeline and not included directly by any project source. Full local verification (mandatory for the final target): fresh configure + full cmake --build (libjllama.so + jllama_test compile and link, -O3, no undefined references) + ctest 482/482 passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv
bernardladenthin
had a problem deploying
to
maven-central
August 7, 2026 18:32 — with 
GitHub Actions
Failure
GitHub Actions
Failure
bernardladenthin
had a problem deploying
to
startgate
August 7, 2026 18:32 — with
GitHub Actions
Error
GitHub Actions
Error
bernardladenthin
had a problem deploying
to
maven-central
August 7, 2026 18:32 — with
GitHub Actions
Failure
GitHub Actions
Failure
|
This PR comprehensively migrates TTS from OuteTTS to Qwen3-TTS. The code is well-structured, properly error-handled, and thoroughly tested. Two minor documentation suggestions: (1) clarify that the default language is 'english' in the Java docstring, (2) mention supported audio formats for speaker reference files. No security concerns. Ready to merge. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| inp.prompt = text.c_str(); | ||
| inp.prompt_len = text.size(); | ||
| inp.speaker_ref = speaker_bitmap.get(); | ||
| inp.lang = lang.empty() ? "english" : lang.c_str(); |
There was a problem hiding this comment.
The default language "english" is hardcoded here, but the Java docstring for synthesize() doesn't specify this. Consider clarifying in the Java Javadoc that when language is null/empty, "english" is used as the default.
Also consider documenting what language values the upstream Qwen3-TTS model supports (the comment references tools/tts/README.md but that's not directly linked).
Comment on lines
+82
to
+90
| * Synthesize speech with an optional cloned voice and language. | ||
| * | ||
| * @param text the text to speak | ||
| * @param maxCodeTokens cap on generated audio-code tokens (longer = longer audio) | ||
| * @param topK top-k sampling cutoff for the code model | ||
| * @param speakerReferenceAudioPath path to a reference audio clip (wav/mp3) whose voice is | ||
| * cloned, or {@code null}/empty for the model's default voice | ||
| * @param language ISO 639-1-ish language name understood by the model (e.g. {@code "english"}, | ||
| * {@code "chinese"} — see the model's own documentation for the supported set), or | ||
| * {@code null}/empty for the model's default | ||
| * @param maxFrames cap on generated audio frames (longer = longer audio) |
There was a problem hiding this comment.
This docstring could be more explicit about what formats are supported for the speaker reference audio. Consider adding a note like: "(supported formats: WAV, MP3)" to match the native error handling behavior.
Comment on lines
1790
to
+1793
| const std::string text = parse_jstring(env, jtext); | ||
| const std::string speaker_reference_path = | ||
| jspeaker_reference_path == nullptr ? std::string() : parse_jstring(env, jspeaker_reference_path); | ||
| const std::string lang = jlang == nullptr ? std::string() : parse_jstring(env, jlang); |
There was a problem hiding this comment.
Good defensive programming here: properly handling nullable Java String references and converting them to empty C++ strings. This ensures the native engine can distinguish between null/empty inputs and apply appropriate defaults. ✅
|
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mtmd_helper::gen_audioinstead of hand-copied DSP helpers.cmake/generate-tts-upstream.cmakeandtts_upstream.hare no longer needed; all TTS logic is now upstream-owned via the mtmd library.mtmd_helper::gen_audioAPI; no more manual prompt building, codec filtering, or vocoder embedding-to-audio conversion.TextToSpeechconstructor now takes a backbone model path and mmproj path (instead of TTC and vocoder paths);synthesize()gains optionalspeaker_reference_pathandlangparameters.Test plan
test_tts_wav.cpp— removed OuteTTS codec-token filtering tests (no longer applicable); WAV encoding tests remain and pass.TtsIntegrationTest— updated to use Qwen3-TTS models; exercises the newsynthesize(text, speaker_ref, lang, ...)signature.NativeLibraryLoadSmokeTest— verifies JNI bindings load correctly.cmake --buildwith-O3on b10319 checkout;ctest485/485 passing.publish.ymlupdated to download Qwen3-TTS models instead of OuteTTS/WavTokenizer.Related issues / PRs
docs/history/llama-cpp-breaking-changes.mdfor the b10269→b10270 architecture transition details.Checklist
CONTRIBUTING.mdandCODE_OF_CONDUCT.mdhttps://claude.ai/code/session_01QyzMNtQf878jjtLc6KCxVv