Xuejun/enable npu opt v1 - #305
Open
zhaixuejun1993 wants to merge 23 commits into
Open
Conversation
Static shapes: - get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic ([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through GET_ROWS into the conv/GDN state, the state reshapes and the GDN output. - With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and skip binding its zero-byte ggml tensor as an output (the dynamic path already did the latter, the static path wrote the full cache over a 0-byte buffer). Token-count independence: - In static mode the compiled model's token count is the prefill chunk size or 1, not the captured cgraph's. Offsets derived from the captured count were therefore wrong. Anchor the GDN state slice at the end of the packed [attn | state] output and drop the rs_src_begin runtime inputs, and make VIEWs over the GDN output / conv_input pass through so the consumer does the slicing. - CONT could not identify its token axis when the graph was captured with a single token (every trailing dim has the same stride and size 1) and baked the captured shape into the prefill model. Chunked prefill: - The last chunk is padded with fabricated tokens. Attention masks them, but the recurrent path folded them into cache_r/cache_s permanently. Add a chunk_valid_len runtime input, use it to zero g and beta for padded steps (making the recurrence an exact identity) and to end the conv snapshot window at the last valid token, and disable the recurrent-cache reset after the first chunk so earlier chunks are not wiped. - get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but IMROPE stacks 4 position planes, so every decode step was run through the padded prefill model and the loop ran extra out-of-bounds chunks. cache_rs_reset_idx/len now stay runtime Parameters in static mode, since can_reuse_statically() does not invalidate the cached model on ComputeParams changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU.
Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch. When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged. Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table.
Groundwork that was sitting uncommitted in the tree, kept separate from the tied-embedding change that follows: - env passthrough for NPU_COMPILER_TYPE, NPUW_FUNCALL_FOR_ALL, NPUW_UNFOLD_IREQS and COMPILATION_NUM_THREADS, plus a generic GGML_OPENVINO_NPU_CONFIG KEY=VALUE escape hatch for bisecting plugin options without a rebuild - GGML_OPENVINO_KV_SCATTER_ELEMENTS: ScatterElementsUpdate instead of ScatterUpdate for the single-row KV write, which is not in-place on NPU - GGML_OPENVINO_REDUCE_COMPILE_MEM: stream requantization in chunks rather than materializing the whole tensor - GGML_OPENVINO_RELEASE_WEIGHTS / GGML_OPENVINO_COMPILE_FROM_IR fail-fast guards for recompiles after host weight buffers are dropped - GGML_OPENVINO_TOKEN_EMBD_I8 and GGML_OPENVINO_NPU_KEEP_Q4_0 opt-ins
For a tied embedding the token_embd table is also the lm_head weight, so one OV constant feeds both a GET_ROWS at layer 0 and a MUL_MAT after the last layer. Those two consumers land in different NPUW partitions, and a value crossing a partition boundary has to be materialized, so gathering rows out of the dequantized table forces the whole table into real memory. That is why compressing the table used to lose: the dequantization traffic cost more than the smaller weights saved. gather_compressed_rows() walks the dequantization chain emitted by make_int4_weights/make_int8_weights, gathers the requested rows out of the leaf Constants, and rebuilds the chain on top of them. The row lookup then reads packed rows and dequantizes only those, and the chain feeding the lm_head matmul is left with a single consumer, so it stays compressed. Only the node types those two builders emit are recognized; anything else falls back to the plain Gather, and the rewrite is skipped for tables under 1M elements and for chains with no low-precision Constant, which leaves the current f16 default bit for bit unchanged. GGML_OPENVINO_TOKEN_EMBD_I4 opts token_embd into group-128 int4, which is only worth doing together with the above. phi4-mini Q4_0 on Panther Lake, -p 512 -n 32 -b 512 -ub 512: arm pp512 tg32 device p50 f16 (default) 344.34 10.70 89.71 ms int4 + packed gather 346.17 13.11 70.59 ms int4, gather not sunk 328.97 8.26 118.11 ms Decode weight traffic goes from 2.90 GB to 1.99 GB per step, and the measured 19.1 ms saving matches the 21.5 ms that predicts at the 42.4 GB/s the device reaches on an f16 matmul of this shape. Accuracy of the int4 embedding is NOT established. llama-perplexity cannot run on this backend (it asks for all-position logits, which the static NPU graph has no shape for), and the one greedy sample that could be compared diverged and read slightly worse. The flag stays opt-in for that reason.
Allow the Q/K ROPE subviews used by Phi-4 so the model remains a single OpenVINO graph, avoiding NPUW f16 interconnect output mismatches. Serialize prefill and decode static model compilation because concurrent NPUW compilation is unstable.
ravi9
force-pushed
the
dev_backend_openvino
branch
2 times, most recently
from
September 3, 2026 21:10
fd9bc04 to
33237ab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several enhancements and configuration options to the OpenVINO integration in
ggml, primarily focused on improving NPU (Neural Processing Unit) support, quantization flexibility, and memory efficiency. The changes expand environment variable controls, add new quantization pathways, and optimize certain operations for better performance on NPUs.Key changes include:
Expanded NPU and Quantization Configuration
Added support for new environment variables to fine-tune NPU compilation and runtime behavior, including
GGML_OPENVINO_NPU_COMPILER_TYPE,GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL,GGML_OPENVINO_NPUW_UNFOLD_IREQS,GGML_OPENVINO_COMPILATION_NUM_THREADS,GGML_OPENVINO_NPU_CONFIG,GGML_OPENVINO_TOKEN_EMBD_I8,GGML_OPENVINO_TOKEN_EMBD_I4,GGML_OPENVINO_NPU_KEEP_Q4_0,GGML_OPENVINO_COMPILE_FROM_IR, andGGML_OPENVINO_KV_SCATTER_ELEMENTS. These allow more granular control over compilation strategies, quantization types, and memory/performance trade-offs. [1] [2] [3]Enhanced quantization logic for embedding tables, allowing selection between F16, int8, or group-128 int4 representations based on new environment variables. This enables memory and performance optimizations, especially for tied embeddings.
Quantization and Memory Optimizations
Updated the
quantize_q4_0function and its usage to support streaming quantization with correct block offset handling, enabling efficient chunked quantization and reducing peak memory usage during compilation. [1] [2] [3] [4] [5] [6] [7]Modified the
get_rowsoperation to gather compressed (packed) embedding rows when possible, avoiding unnecessary materialization of large dequantized tables and improving efficiency for tied embeddings.NPU-Specific Operation Improvements
ScatterElementsUpdateinstead ofScatterUpdatefor key-value cache updates on NPU, reducing unnecessary data copying and improving performance for single-row writes. [1] [2]Bug Fixes and Code Cleanups
Fixed a bug in ROPE operation support checks, ensuring correct dimension comparisons for view operations.
Included missing header for extra OpenVINO helpers in
set_rows.cpp.## OverviewAdditional information
Requirements