Sync master with upstream release b10502 - #626
Open
jan-service-account wants to merge 147 commits into
Open
Conversation
* Switch ROCm from 7.2.1 to 7.14 ROCm 7.14 is the first production release using TheRock build system. It can be installed using multi-arch deliverables from wheels, debs, rpms, tarballs or runfiles. Adjust ROCm targets for Linux and Windows to use this instead. * ci: switch all other Windows ROCm jobs to ROCm 7.14 wheels Move the shared windows-setup-rocm composite action from the HIP SDK PRO Edition installer to the multi-arch ROCm wheels (rocm[libraries,devel]). The wheel-install logic that previously lived inline in release.yml is now in the shared action, and both build-cache.yml and release.yml call it. Also migrate the build-cuda-windows.yml hip job to the same wheel-based layout (cache path/key, rocm-sdk environment setup, llvm/bin compiler paths) so it keeps working after the action's contract changed; drop its now-unused ROCm 7.2.1 rocWMMA download and stale include path.
…ml-org#26566) * test new flash_attn test * rebase and fix to disable subgrou matrices when max_kv_tile == 0 * delete log output * Add i32 support to cpy and enables the all ops test * restore the non target ci tests * comment out of TODO of build-cpu.yml * fix format
* model : fix SWA not being enabled for EXAONE 4.5 load_arch_hparams tests `hparams.n_layer() == 64` before LLM_KV_NEXTN_PREDICT_LAYERS has been read. n_layer() returns n_layer_all - n_layer_nextn and n_layer_nextn defaults to 0, so a GGUF carrying the MTP head (block_count=65, nextn=1) evaluates to 65 and the whole SWA block is skipped. The model type switch further down in the same function reads 64, because by then the key has been loaded. n_swa is still filled in by the unconditional get_key below the block, so llama_model_n_swa() reports 4096 and the logs look correct while only swa_type stays LLAMA_SWA_TYPE_NONE. This affects the official LGAI-EXAONE GGUF release as well. EXAONE 4.0 has no MTP head, so block_count is 64 there and the check matches. * model-loader : skip TENSOR_SKIP tensors in the metadata-only path create_tensor asserts on a null buffer type when building from metadata alone, but buft_for_tensor returns null by design for tensors marked TENSOR_SKIP, which is how architectures with nextn/MTP layers mark theirs. Those models cannot be constructed by llama_model_init_from_user at all. The file-backed path below already returns nullptr for the same tensors, so callers see the same thing either way. * tests : cover exaone4 hparams ordering Builds a synthetic exaone4 model with the layout the shipped EXAONE 4.5 GGUFs use (block_count 65 + nextn 1). The swa_type check is the one that catches the ordering bug; the n_layer_nextn and n_layer() checks only tell a broken fixture apart from a real regression. Fails before the ordering fix with "swa_type is not STANDARD", passes after. * Revert "tests : cover exaone4 hparams ordering" This reverts commit d2f3baf. * Revert "model-loader : skip TENSOR_SKIP tensors in the metadata-only path" This reverts commit aecb9bc.
* test-backend-sampler: skip multi_output_sampling_chain on HIP The new multi_output_sampling_chain test uses top_k, whose backend probs path needs CUB (unavailable on HIP), so sampled_probs is null and the test aborts. Add it to the existing HIP skip list alongside the other TOP_K tests. * ci: keep gpu-rocm logs in a per-run dir keyed by GitHub run id The self-hosted gpu-rocm runner can't upload logs to Azure blob (egress firewalled), so a run's logs were wiped by the next run. Write each run's logs to $OUT/run-<run_id>-<attempt>/ so an Actions run URL maps to its logs. * test-backend-sampler: also skip multi_output_cpu on HIP Like the other TOP_K-based subtests, multi_output_cpu's backend sampler never initializes on HIP (no CUB TOP_K), so it aborts. Add it to the skip list. --------- Co-authored-by: Jim Wu <ywu@xilinx.com>
* tests : remove fetch_server_test_models.py * ci : use tests.sh wrapper of pytest
…rg#26081) * llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * set mmap_support to false on OpenCL backend * fix order of load modes * use -1 for auto * resolve load mode auto earlier to correctly pick gpu host or cpu memory * add load mode auto to llama-bench * bump virtgpu api version, regenerate docs --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml-org#26890) This commit updates the python script that runs the original model to generate embeddings for the causal model, to use save_output_data which stores the token ids and the prompt in addition to logits. The motivation for this is that the embedding logits verification will fail as it expects these files (-prompt.txt and -tokens.bin) to exist. With the changes in this commit the causal-verify-embeddings target works again.
Most of the old ones have been resolved (yay) but the recent refactor of mmq paramters has caused some symbol names to change, leaving a couple of non-ignored failures
* webui: hide loaded model in context gauge at single-model mode * webui: keep context gauge details open state across reopens
* adapt the api * text model ok * working impl, need verify and clean up * mtmd: build the pocket-tts transposed convolutions as GEMM + col2im ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts. * flow_temp + frames_after_eos * chunking * mtmd: carry the remaining pocket-tts per-pack settings The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence. * clip.gen.audio.model_variant * clean up code comments * nit: drop the dead flow_temp hparam, the pack table holds the default * update docs * address security problems * less invasive base.py * lint * add mtmd_gen_inp_default * add docs * rm gen_flow_temp --------- Co-authored-by: Pascal <admin@serveurperso.com>
* conversion: skip untrained DFlash embeddings * Add Nemotron DFlash support * Add DFlash NVFP4 support * Address review comments * add missing output_s for nvfp4 * Include change for keeping residual for last layer also if requested in future dflash models * Update conversion/qwen.py Defensive check, not needed Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Fixing bug introduced by merge conflict --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…g#26903) Signed-off-by: ynankani <ynankani@nvidia.com>
* spec : update speculative-simple * cont : simplify * cont : clean-up
…6879) * chat : fix muse-glimmer swallowing a trailing tool call into content Muse Glimmer routinely answers the user and calls a tool in a single generation. The template terminates a message with <|eom|> when more messages follow in the same turn and <|eot|> only at the end of the turn, so the answer is closed by <|eom|> and the call opens a fresh header: <prose><|eom|><|start|>assistant to=<tool><|message|><atem:function_calls>... The final-message rule read content with until("<|eot|>"), which assumed the user-facing message is always last. There is no <|eot|> before the call, so content ran to the end of the turn, absorbed the markup, and no tool_calls were emitted - the tool never ran. On a tau2-bench telecom run this hit 43 turns across 19 of 114 tasks. Stop the answer at <|eom|> and parse what follows as tool calls. Adds models/templates/muse-glimmer.jinja and four parser tests: a plain answer, the <|eom|> junction, markup quoted in an answer staying content, and tool markup inside the to=self channel staying reasoning. * address comment
…-org#26882) * fix: handle nested global_head_dim in Gemma4 config Gemma-4 E4B models have global_head_dim inside text_config rather than at the top level. Add fallback to support both layouts. * fix: add fallback for global_head_dim to support per_layer_config format * fix: read head_dim only from full_attention layers in per_layer_config and num_global_key_value_heads compatibility * fix: added fallback for num_global_key_value_heads * fix: read per_layer_config from root hparams * fix: delete unused text_config * cleanup and fixes --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…all (ggml-org#26892) * wavtokenizer-dec : bound posnet/convnext block_count against n_layer_all * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12 logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not wired yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * tests: enable TQ2_0 in backend-ops type lists Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends without support skip via not-supported as usual. TQ1_0 stays disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* mtmd : skip thumbnail for non-tiled LFM2 images Matches HF behavior. * Update tools/mtmd/mtmd-image.cpp --------- Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* mtmd: use sha256 for input hashing * void conflict with boringssl
…E [no release] [no ci] (ggml-org#26847) * doc: document MCP stdio servers and CORS defaults in the server README The MCP arguments were listed but nothing explained what an MCP server is or how to declare one. Cover the stdio transport, the config keys, the tool naming, and add a POSIX shell echo server as a minimal example. Also document the CORS behavior: the default reflected origin, the switch to localhost once tools are enabled, and the recommended setting per deployment. * doc: drop the inline MCP shell example from the server README The example parsed JSON-RPC by hand and sat in a page people copy paste from, into servers spawned with the privileges of llama-server. Point to the specification instead. Link the pull request that introduced the feature, and keep a short mcp.json snippet so the table of configuration keys has a declaration to refer to.
* replace rpc job with cpu * remove vulkan cache * move windows to build vulkan
* server: rename built-in tools to server tools * ui: rename built-in tools to server/browser tools
…26843) * CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark Signed-off-by: ynankani <ynankani@nvidia.com> * skip moe experts and allow others based on k geometry (allow only small idle tail) Signed-off-by: ynankani <ynankani@nvidia.com> * rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture Signed-off-by: ynankani <ynankani@nvidia.com> --------- Signed-off-by: ynankani <ynankani@nvidia.com>
…ase (ggml-org#27302) * ci : create pre-release with change log and nightly link in make-release After pushing the tag, create a pre-release using ggml-org/action-create-release. The release description is generated by scripts/make-release-desc.sh: the change log between the current and previous version (one line per commit), a link to the corresponding nightly build when it exists, and a note that semantic versioning is still work in progress. Assisted-by: pi:llama.cpp/Qwen3.8-27B * cmake : bump version to 0.1.2 Assisted-by: pi:llama.cpp/Qwen3.8-27B * ci : find the nightly tag by commit in make-release-desc.sh The nightly release is guaranteed by the release checks to point at HEAD, so instead of reconstructing its name (commit count, branch, hash) just pick the b* tag pointing at HEAD. This also drops the RELEASE_BRANCH env var from the workflow. Assisted-by: pi:llama.cpp/Qwen3.8-27B * ci : resolve the release commit from the version tag in make-release-desc.sh The change log and nightly lookup now use the commit the version tag points at (HEAD when the tag does not exist), instead of always HEAD. This makes the script usable locally for older versions, e.g. ./scripts/make-release-desc.sh v0.1.1. The tag is resolved to a SHA first, since --points-at does not peel annotated tags. Assisted-by: pi:llama.cpp/Qwen3.8-27B * ci : normalize the version argument in make-release-desc.sh Accept the version with or without the leading v (0.1.1 == v0.1.1) and reject anything else, instead of silently treating a bare version as a non-existent tag name. Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : clean-up
* xcframework : fix build * mtmd : remove unused include path * vendor : use vendor::hash alias target in cmake CMake reserves "::" in target names for imported/alias targets, so the real target keeps the name vendor-hash and a vendor::hash ALIAS target is added. Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias. Assisted-by: pi:llama.cpp/Qwen3.8-27B * vendor : add cmake targets for all vendored libs with vendor:: aliases Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann, sheredom, stb) and ALIAS targets named vendor::<lib> for all of them, including cpp-httplib and hash. Each exposes the vendor/ root so includes are namespaced, e.g. <nlohmann/json.hpp>. Consolidate the per-lib add_subdirectory calls into a single add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON. Consumers (llama-common, mtmd) now link the aliases instead of relying on raw vendor/ include paths. hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept as a PRIVATE include so the synced upstream sources compile unmodified. Assisted-by: pi:llama.cpp/Qwen3.8-27B * readme : use foo/bar names in acknowledgements Assisted-by: pi:llama.cpp/Qwen3.8-27B * ocd : fix valign
* mtmd: fix LFM2 image tiling threshold * refactor testing * fix * fix on windows --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
…7300) - Add a CUDA 13.4 ARM64 matrix entry. - Build only ggml-cuda for x64 and ARM64.
…rg#27292) * update to ov-2026.3, update device drivers * ci: skip nemotron-h rollback test on OpenVINO The OpenVINO backend does not support SSM_SCAN, so the Nemotron-H recurrent state rollback graph is split and cannot preserve the recurrent cache output shape. Keep the test enabled for other backends and retain the qwen35 OpenVINO rollback coverage. --------- Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
* readme : update status badges * authors : regen
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
Kernel is a port of `ggml-cuda/fwht.cu` (us/run, median): ``` m x n x k GEMM FWHT speedup 64 x 1 x 64 10.20 2.93 3.48x 64 x 2048 x 64 10.75 2.71 3.97x 128 x 1 x 128 10.33 2.88 3.59x 128 x 32 x 128 9.20 2.77 3.33x 128 x 2048 x 128 16.46 2.76 5.95x 256 x 1 x 256 10.19 2.77 3.68x 256 x 2048 x 256 16.69 3.41 4.89x 512 x 2048 x 512 54.16 12.89 4.20x ```
…ch layer (ggml-org#22877) * Evict weights from memory after processing each layer * Revert changes * Move unmap to libllama * Unmap weights offloaded to backend * Change member's constness * Remove unmap weights offloaded to backend
* ui: Remove dead code from stores - persisted() helper was exported but never used - messageUpdateCallback / registerMessageUpdateCallback were never wired up - conversationsStore.initialize() alias, single caller moved to init() * ui: Merge device, theme and viewport into a single deviceStore All three are reactive browser-environment signals, now exposed as one class store: deviceStore.isMobile, deviceStore.isIOSDevice / isIOSSafari / isWKWebView / isStandalone and deviceStore.systemTheme.isDark. The systemTheme name disambiguates the OS preference from the user theme preference in settingsStore. Drops the unused viewport export (only isMobile was consumed). * ui: Merge build info into version store One VersionStore class with build (llama.cpp build number from build.json) and frontend (PWA version from _app/version.json), matching the class pattern of the other stores. * ui: Colocate context gauge popup state with its components The gauge popup state is local UI state shared only by the ChatFormContextGauge subtree, so it lives next to its consumers instead of the app-scope stores barrel.
* ui: Move stream lookup and replay fetches into ChatService chatStore called fetch() directly for /v1/streams/lookup and the /v1/stream replay. These now live next to the other stream-session methods in ChatService, so services stay the only API I/O layer. * ui: Move /models/sse feed reader into ModelsService ModelsService.watchModelEvents owns the byte stream, reconnect loop and SSE record parsing; modelsStore keeps only event routing and state. * ui: Extract conversation import/export into ConversationTransferService The JSONL session format, ZIP archiving and browser downloads are pure I/O with no store state, so they move out of conversationsStore. The store keeps the DB orchestration (bulkExportConversations, downloadConversation, importConversationsData) and delegates the format work. * ui: Consolidate active model resolution into modelsStore.activeModelId The same resolution chain was duplicated in useChatScreenActiveModel, ChatForm, ChatFormActionModels and contextStatsStore, with slight drift in the single-model fallback. The canonical getter now lives in modelsStore, and the shared last-assistant-model lookup moved to utils as getConversationModel. * ui: Initialize stores explicitly via initStores() Store constructors and module-level side effects ran migrations and localStorage reads in import order. Migrations rename and rewrite localStorage keys, so a settings load racing ahead of them could clobber migrated values. initStores() is called once from the root layout and runs migrations first, then the stores that read localStorage, then the conversations DB load. * refactor: Constants for stream query params
* gguf-py : add size guards to GGUFReader Guard kv_count, tensor_count, string length, and array length against crafted values that cause unbounded allocation or hangs. Assisted-by: opencode * gguf : validate tensor data section fits within file When no_alloc=true, gguf_init_from_reader accepted files where the tensor data section (computed from header claims) exceeded the remaining file size. This allowed crafted GGUF files to pass validation while having insufficient data, leading to OOB reads when the loader later mapped tensor data from the file. Assisted-by: opencode * gguf-py : move size limits into gguf_reader.py Per review feedback, the limits are not part of gguf.h but are arbitrary limits defined in gguf.cpp, so define them locally in the reader instead of exporting them from constants. Assisted-by: opencode * remove gguf.ccp changes --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates dev branch with latest release (b10502) from ggml-org/llama.cpp