Sync with Microsoft ONNX Runtime - 09082026 - #1252
Closed
ai-fw-intg wants to merge 30 commits into
Closed
Conversation
--- updated-dependencies: - dependency-name: SixLabors.ImageSharp dependency-version: 2.1.11 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…icrosoft#31635) ## Description Extends the CUDA plugin EP packaging pipeline with Windows ARM64 support, replaces the shared x64/ARM64 architecture lists with per-platform lists that reflect the GPUs each package actually targets, and adds two build-time knobs for controlling binary size. The plugin `.so`/`.dll` is currently very large (~720 MB uncompressed on the CUDA 12.8 Linux leg, ~90% of which is `.nv_fatbin`), and the architecture list is the single biggest lever on that number, so it needs to be tuned per platform rather than shared. ## Summary of Changes ### Windows ARM64 packaging | File | Change | |------|--------| | `plugin-cuda-pipeline.yml` | Add `build_windows_arm64`; rename `invalidAArch64Config` to `invalidArm64Config` and extend it to cover Windows ARM64, since NVIDIA only ships Windows-on-ARM CUDA for 13.x | | `plugin-cuda-packaging-stage.yml` | Add `build_windows_arm64` and `arm64_cuda_version` (13.1), wire the ARM64 stage and its artifacts into NuGet and Foundry Local zip packaging | | `plugin-win-cuda-stage.yml` | Add an `arm64` arch path: ARM64 agent pool, `win-arm64/` CUDA SDK blob prefix, separate cuDNN folder, native ARM64 toolset | ### Per-platform CUDA architecture lists `cmake_x64_cuda_archs` / `cmake_arm64_cuda_archs` are split into four independent lists, since Windows x64, Linux x64, Windows ARM64, and Linux aarch64 serve very different GPU populations: | Parameter | CUDA 12.8 | CUDA 13.x | |------|------|------| | `cmake_windows_x64_cuda_archs` | `61,75,86,89,120` | `75,80,86,89,120` | | `cmake_windows_arm64_cuda_archs` | n/a | `120,121` | | `cmake_linux_x64_cuda_archs` | `75,80,86,89,90,120` | `75,80,86,89,90,120` | | `cmake_linux_aarch64_cuda_archs` | n/a | `89,90,100,103,120,121` | Notable decisions: - **`120-virtual` dropped everywhere.** The `compute_120` PTX measured 176 MB, 27% of the entire `.nv_fatbin` — by far the most expensive single entry. It also cannot carry the NVFP4 kernels, which are only valid as real `sm_120a` (`cuobjdump -ptx | grep -c e2m1x2` returns 0), so it was paying full price for partial coverage. - **Linux aarch64 targets the platforms that actually exist on ARM**: GH200 (`90`), GB200 (`100`), GB300 (`103`), DGX Spark GB10 (`121`), plus discrete cards in ARM chassis (`89`, `120`). `103` is required alongside `100` because ORT normalizes `100` to `100a-real`, and `a` targets are locked to their exact SM. - **`75` dropped from Linux aarch64** — Turing was never paired with an ARM host in practice. ### CUDA architecture normalization | File | Change | |------|--------| | `cmake/external/cuda_configuration.cmake` | `ARCHITECTURES_WITH_ACCEL`: add `103` and `121`, drop `101` (removed by NVIDIA after CUDA 12.9). Without this, `103` and `121` would be built as plain targets and would silently lose the CUTLASS block-scaled/TMA kernels, which are gated on `__CUDA_ARCH_FEAT_SM1xx_ALL`. | ### Build size controls Two new pipeline parameters, both plumbed through the packaging stage to all four platform stages: | Parameter | Default | Effect | |------|------|------| | `enable_cuda_fatbin_size_compression` | `false` | Sets the new `onnxruntime_CUDA_FATBIN_COMPRESS_SIZE` cmake option, forcing `-Xfatbin=-compress-all -compress-mode=size` on the CUDA 12.8 leg. CUDA >= 13.0 already does this unconditionally, so the parameter only changes 12.8. | | `enable_fpa_intb_gemm` | `true` | Sets the existing `onnxruntime_USE_FPA_INTB_GEMM` cmake option. fpA_intB GEMV/GEMM is ~141 MB of device code (22% of `.nv_fatbin`), second only to flash attention. | `onnxruntime_CUDA_FATBIN_COMPRESS_SIZE` fails configuration on CUDA < 12.8 rather than silently passing an unsupported flag to nvcc. Windows composes these via `FatbinCompressOption` / `FpaIntBGemmOption` job variables appended to the `build.py` invocations, mirroring the existing `$(TelemetryOption)` pattern. Linux composes them into `EXTRA_CMAKE_DEFINES`, which `build_cuda_plugin_package.sh` already forwards. ### Packaged binary hardening and verification | File | Change | |------|--------| | `cmake/onnxruntime_providers_cuda_plugin.cmake` | Compile `onnxruntime_providers_cuda.rc` into the plugin DLL on Windows so the packaged binary carries version info; set `SKIP_BUILD_RPATH` on Linux so the build machine's CUDA path is not embedded in a binary that ships as-is | | `plugin-linux-cuda-stage.yml` | Fail the build if the plugin `.so` has an empty `RPATH`/`RUNPATH` component or a hard-coded CUDA path | | `plugin-win-cuda-stage.yml` | Fail the build if the plugin DLL is missing required version-info properties | ## Testing - Pipeline changes are validated by running the CUDA plugin packaging pipeline. Both new parameters default to current behavior (`enable_cuda_fatbin_size_compression: false`, `enable_fpa_intb_gemm: true`), so a default run produces the same build flags as before this PR aside from the architecture list changes. - The cmake `-compress-mode` selection logic was verified in isolation across four combinations: | Toolkit | Option | Result | |---|---|---| | 12.8 | OFF | `-Xfatbin=-compress-all` | | 12.8 | ON | `-Xfatbin=-compress-all -compress-mode=size` | | 13.1 | OFF | `-Xfatbin=-compress-all -compress-mode=size` | | 12.6 | ON | configure-time fatal error, as designed | - The new RPATH and DLL version-info checks are self-verifying: they fail the packaging stage rather than publishing a bad artifact. ## Motivation and Context The primary consumer is Foundry Local (vision, audio, and mostly LLM models), which ships this plugin to end-user machines, so download size matters directly. Trade-offs worth flagging for reviewers: - **`-compress-mode=size` raises the minimum driver** to the CUDA 12.4 level (Linux >= 550.54.14, Windows >= 551.61); older drivers cannot decompress the fatbin at all. It also increases module load time (measured ~0.8 ms to ~4.3 ms for a ~6.5 MB SASS module) and adds a few percent to nvcc time. This is why the parameter defaults to `false` and is opt-in per run. - **The architecture lists are all `-real` with no virtual entry**, so any GPU whose compute capability is not explicitly listed gets `cudaErrorNoKernelImageForDevice` (209) instead of falling back to JIT. This is deliberate given the PTX cost, but it means new architectures must be added explicitly. - **`enable_fpa_intb_gemm: false` is not yet validated end to end.** The fpA_intB path is opt-in at run time via `ORT_FPA_INTB_GEMM` / `ep.cuda.fpa_intb_gemm`, but `matmul_nbits.cc` forces it on whenever weights are prepacked, independent of that flag. The fallback path should be exercised before shipping a package built with this off. ## Checklist - [x] No breaking changes to default pipeline behavior (both new parameters default to existing behavior) - [x] cmake option gated on toolkit version with an explicit error rather than a silent no-op - [ ] Tests added/updated — not applicable; changes are build/packaging configuration
…nd head sink (microsoft#29912) ### Description `PagedAttention` is ORT's continuous-batching attention operator, but on `main` it only supports FP16/BF16 caches with RoPE and softcap, has no paged decode kernel, and forces a device→host synchronization on every node on every step (which makes it uncapturable by CUDA graphs). This PR brings it to feature parity with `GroupQueryAttention` for the popular LLM families and adds the paging and latent-cache primitives that serving frameworks need, **additively** — every model valid under the shipped `com.microsoft::PagedAttention` opset-1 schema keeps working unchanged. The design rationale, the compatibility invariant, and the alternatives that were considered and rejected are written up in the new design document [`docs/contrib_ops/cuda/paged_attention.md`](docs/contrib_ops/cuda/paged_attention.md); the section numbers referenced below point into it. ### Summary of Changes #### Schema (`bert_defs.cc`, `docs/ContribOperators.md`) All additions are trailing optional inputs, new attributes whose defaults reproduce current behavior, or widened type constraints (§4). | New input | Idx | Purpose | |---|---|---| | `slot_mapping` | 10 | Explicit per-token cache slot, so the scheduler owns placement instead of the kernel re-deriving it (§5) | | `head_sink` | 11 | Attention sink / smooth softmax, matching GQA (§6) | | `q_norm_weight` / `k_norm_weight` | 12, 13 | Fused QK-RMSNorm (Qwen3, gpt-oss) (§7) | | `k_scale` / `v_scale` | 14, 15 | Per-tensor or per-channel dequantization scales for a quantized cache (§8) | | `attention_metadata` | 16 | Optional CPU input carrying *replay-wide upper bounds* `[max_query_len, max_kv_len]`, which removes the per-node per-step D→H sync (§4.7) | | New attribute | Default | Purpose | |---|---|---| | `qk_norm_epsilon` | `1e-6` | Epsilon for the fused QK-Norm | | `k_quant_type` / `v_quant_type` | `NONE` | `NONE` \| `PER_TENSOR` \| `PER_CHANNEL` | | `k_cache_dtype` / `v_cache_dtype` | `""` | Logical cache element type, named after the ONNX type it denotes | | `kv_cache_layout` | `SEPARATE` | `SEPARATE` \| `LATENT` (absorbed MLA: one cache, no `value`/`value_cache`) | | `v_head_size` | `0` | Narrower V head, `LATENT` only (DeepSeek-V3 uses 576/512) | | `rotary_offset` | `0` | Applies RoPE to `[rotary_offset, rotary_offset + rotary_dim)` so MLA can rotate only the positional suffix | `key_cache` / `value_cache` move from `T` to a new `T_CACHE` constraint (`float16`, `bfloat16`, `int8`, `float8e4m3fn`), and `value_cache` / `value_cache_out` become optional so a `LATENT` node can omit them. Shape inference now takes the cache element type from inputs 3/4 rather than from `query`, which was wrong for a quantized cache. #### CUDA kernels (`paged_attention_impl.cu`, `paged_attention.cc/.h`, `paged_attention_helper.h`) - **Paged decode kernel** (`LaunchPagedDecodeAttention`) — split-KV, block-table-aware decode with native head-sink, softcap, sliding-window and on-the-fly cache dequantization. - **XQA paged decode** (`onnxruntime/contrib_ops/cuda/bert/xqa/`) — TensorRT-LLM's XQA kernels extended to the paged block layout: 8 new translation units `xqa_paged_{fp16,bf16}_{int8,fp8}_{64,128}.cu` plus a shared paged loader. Selected for quantized-cache decode. - **Quantized paged cache** — `ReshapeAndCache` quantizes on write; all read paths dequantize with `k_scale`/`v_scale` under `PER_TENSOR` or `PER_CHANNEL` granularity. - **`ApplyHeadSink`** — exact post-hoc LSE rescale (`1/(1+exp(s_h − lse))`) applied *after* the quantized/unquantized branch, so no backend can silently drop the sink (§6). - **`QkNormRotaryTNH`** — fuses QK-RMSNorm, RoPE (with `rotary_offset`) and the packed-QKV unpack into one pass. - **Absorbed MLA** (`PagedLatentAttentionKernel` / `LatentAttention`) — single latent cache, V read as the leading `v_head_size` channels of the same row that supplies K (§12). - **CUDA-graph safety** — backend dispatch, grid sizing and workspace extents now come from static shapes and the `block_table.shape[1] * block_size` capacity bound; per-step quantities are read on device. The unconditional `cudaStreamSynchronize` is gone from the capturable path (§4.7). - **`int8`→`fp16` conversion fast path** (`xqa/utils.cuh`, `cvtS8x4ToF16x4`) — replaces a scalar `I2F` loop with a `prmt` + `sub.f16x2` sequence (5 full-rate instructions per 4 elements, bit identical). Shared with the non-paged GQA loader. - Kernel registration is now `<T, T_CACHE>`-typed; FP8 combinations are behind `USE_FP8_KV_CACHE && !DISABLE_FLOAT8_TYPES`. #### GQA bug fix (`flash_api.{h,cc}`, `group_query_attention_impl.cu`) `mha_fwd` had `constexpr void* head_sink = nullptr;` hardcoded inside it, and `FlashAttentionAndQuantizeKV` — the *only* GQA prompt path taken when the KV cache is quantized — called it. So for gpt-oss with an INT8/FP8 KV cache, the attention sinks were silently dropped for the entire prompt on every layer while decode stayed correct. `mha_fwd` now takes `head_sink` and GQA forwards it. Op-level prefill error drops **0.074829 → 0.000122**. On gpt-oss-20b (int4 body, INT8 per-channel KV), MMLU-Pro-800 goes **0.6175 (494/800) → 0.7200 (576/800)**. Existing CI missed this because `atol["int8_fp16"] = 1e-1` in `test_gqa.py` is ~800× wider than the post-fix error. #### Tooling and docs - `symbolic_shape_infer.py`: correct output width for packed-QKV and `LATENT` nodes, and cache outputs typed from the cache inputs. - New `docs/contrib_ops/cuda/paged_attention.md` design document; regenerated `ContribOperators.md` and `OperatorKernels.md`. ### Testing `test_paged_attention_cuda.py` grows from a smoke test to ~2k lines / 198 cases, with new suites for features (`slot_mapping`, head sink, QK-Norm), quantized cache (int8/fp8 × per-tensor/per-channel), the paged decode kernel, the XQA decode path, `attention_metadata`, and MLA — each against a PyTorch reference. ```bash python onnxruntime/test/python/transformers/test_paged_attention_cuda.py # 198 passed python onnxruntime/test/python/transformers/test_gqa.py -k xqa # 714 passed ``` The GQA suite is included because the int8 conversion fast path is shared with the non-paged loader. **Backward compatibility.** A node with none of the new inputs/attributes takes exactly the code path it does today: `T_CACHE == T`, `value_cache` present, `kv_cache_layout == SEPARATE`, all quantization `NONE`. The compatibility invariant is stated normatively in §4.2. ### Experimental Results Measured on gpt-oss-20b, H200. E2E numbers are driven through onnxruntime-genai; the CUDA-graph and engine-side plumbing they depend on is **not** part of this PR — they are included to show what the operator-side changes enable, not as a claim about this diff alone. #### Paged decode kernel (isolated, `nh=64 / kvh=8 / hs=64 / block=256`) XQA on/off at `b=8, ctx=4096`, per decode call: | cache | before | after | |---|---|---| | int8 `PER_TENSOR` | 2315 µs | **122 µs** | | int8 `PER_CHANNEL` | 2339 µs | **128 µs** | | fp8 `PER_TENSOR` | 1633 µs | **57 µs** | | fp8 `PER_CHANNEL` | 1566 µs | **57 µs** | Before XQA the quantized paths were ~2.5× *slower* than fp16 — the generic kernel was the bottleneck, not the KV bytes. The `cvtS8x4ToF16x4` conversion path then closes the residual int8-vs-fp8 gap (nsys median, SASS goes from 3928 to 3592 instructions with 192 → 0 `I2F`): | ctx | batch | int8 before | int8 after | gain | fp8 | |---|---|---|---|---|---| | 1024 | 32 | 19.91 µs | 13.60 µs | −31.7% | 12.64 µs | | 4096 | 8 | 24.32 µs | 17.41 µs | −28.4% | 16.48 µs | | 4096 | 32 | 66.62 µs | 42.66 µs | −36.0% | 41.98 µs | | 16384 | 8 | 78.11 µs | 52.58 µs | −32.7% | 49.18 µs | | 16384 | 32 | 244.71 µs | 162.27 µs | −33.7% | 176.70 µs | The int8/fp8 gap goes from up to +59% down to ≤ 7.6% (int8 is faster at the largest config), so the two cache formats can now be chosen on accuracy grounds. #### End-to-end decode throughput (mxfp4 body, INT8 KV, prompt 128 / new 256) | batch | baseline | + `attention_metadata` (no sync) | + CUDA graphs | total | |---|---|---|---|---| | 1 | 242.7 tok/s | 263.8 | **305.7** | **+26.0%** | | 8 | 796.1 | 811.5 | **870.5** | +9.3% | | 32 | 2635.7 | 2682.0 | **2879.2** | +9.2% | Including XQA, batch-1 int8 decode goes **199.2 → 305.7 tok/s (+53.5%)**. After this work attention is no longer the bottleneck at b=1 — the MoE GEMMs and 49 `MatMulNBits` nodes dominate the step, with XQA at 24 × 8.6 µs. #### PagedAttention vs GroupQueryAttention, matched models Two models built from the identical recipe (int4 body, INT8 per-channel KV, identical `num_heads`/`kv_num_heads`/`scale`/window/rotary, byte-identical weight file), differing only in the attention operator. Greedy generation on this stack is bit-reproducible (0/198 discordance across replicates), so there is no sampling noise to subtract. | | GQA | PagedAttention | read as | |---|---|---|---| | MMLU-Pro-800 | 0.7200 (576/800) | 0.7163 (573/800) | +0.4 pp, 3 questions | | GPQA-diamond | 0.6061 (120/198) | 0.6212 (123/198) | −1.5 pp, 3 questions | The two benchmarks disagree in direction and both deltas are 3 questions: **equivalent within noise.** (An apparent +8 pp advantage for paged in earlier runs turned out to be the GQA sink bug fixed above, seen from the other side.) | config | GQA tok/s | paged tok/s | delta | |---|---|---|---| | b=1, p=128, n=256 | 374.4 | **376.3** | +0.5% | | b=2, p=4096, n=256 | 684.3 | **686.6** | +0.3% | | b=8, p=128, n=256 | 1792.7 | 1683.8 | −6.1% | | b=32, p=128, n=256 | 5285.9 | 4109.5 | −22.3% | Peak device memory at matched KV capacity agrees to within 24 MiB (0.16%) from 16k to 128k `max_length` — paged costs nothing extra, and its advantage is structural (a shared pool sized to aggregate demand rather than `batch × max_length`). The b=32 gap was profiled with `nsys --cuda-graph-trace=node`: the captured graph body is at parity with GQA's eager model pass (6.220 ms vs ~6.2 ms) and the entire regression is a 2.384 ms search/sampling tail, which the onnxruntime-genai `Engine` runs once **per request** rather than once per batch. It is not attributable to this operator, and a partial engine-side fix already recovers b=32 to 4517 tok/s. ### Follow-ups (not in this PR) - `attention_bias` and `output_qk` (§10, §11) — schema slots reserved, kernels deferred. - Sub-byte (`int4` / `float4e2m1`) packed caches — attribute vocabulary reserved and rejected at validation until a backend exists (§21.4). - `.Alias(3, 1).Alias(4, 2)` on the kernel def, so a non-aliasing allocation plan fails at partition time instead of run time (§4.4). - Re-tightening `atol["int8_fp16"]` in `test_gqa.py` now that the sink bug is fixed.
### Description <!-- Describe your changes. --> Add an `ort-release-notes` skill with preset configuration, scoped path support, and a concise workflow for metadata generation and draft output. Also include a lightweight docs entry point and WebGPU EP scoped paths file. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Make it easier for AI to generate release note drafts. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Copilot-Session: 3611670d-4e92-4183-8e0e-b2cb659ab624
…t#31632) This change adds a build option for excluding the prebuilt TensorRT fused multi-head attention cubins from the CUDA Execution Provider. The option is enabled by default to preserve existing behavior; disabling it removes approximately 14 MB of embedded cubin data and leaves attention selection to the other available kernels and unfused fallback paths. ## Key Changes - Add `onnxruntime_USE_TRT_FUSED_ATTENTION`, dependent on CUDA and enabled by default. - Propagate the option to CUDA provider compilation and guard TensorRT fused-attention cubin declarations, metadata, and lookup paths. - Filter the prebuilt TensorRT fused-attention cubin sources from both the regular CUDA provider and CUDA plugin provider when the option is disabled, while retaining the shared driver wrapper needed by sparse attention. - Add Windows plugin DLL version metadata and suppress build-machine RPATH embedding for packaged Linux plugin binaries. - Exercise `onnxruntime_USE_TRT_FUSED_ATTENTION=OFF` in the Windows CUDA no-cuDNN plugin build. ## Testing Notes - `git diff --check origin/main...HEAD` passes. - The Windows CUDA no-cuDNN workflow now builds the CUDA plugin with TensorRT fused attention disabled, providing CI coverage for the opt-out configuration. - No local CUDA/Windows build was run in this environment.
This pull request adds input validation checks to prevent integer overflow issues during CUDA kernel indexing in the LayerNorm and RMSNorm CUDA operators. The main goal is to ensure that the product of `num_rows` and `norm_size` does not exceed `INT_MAX`, which could lead to incorrect behavior or crashes. Input validation for CUDA kernel indexing: * Added a check in `LayerNorm::ComputeInternal` (in `layer_norm.cc`) to return an error if `num_rows * norm_size` exceeds `INT_MAX`, preventing integer overflow during CUDA kernel indexing. * Added a similar check in `RMSNorm::ComputeInternal` (in `rms_norm.cc`) to ensure the input size does not exceed CUDA kernel indexing limits. Code maintenance: * Included the `<limits>` header in both `layer_norm.cc` and `rms_norm.cc` to support the new input validation logic. [[1]](diffhunk://#diff-ebda3d3b7054f5d14c679ebe8e6520a2c84a5a558d8fdcc0798c08e7032345feR9) [[2]](diffhunk://#diff-adebea99f15800767eb7b84d40be4873e81ca4df99ffe9dc7f849eabe0a3c563R9) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#31634) ### Description [CPU] Tighten cache_indirection shape contract in MultiHeadAttention ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…fy (microsoft#31159) ### Description Four changes to the fused NVFP4 QMoE decode GEMV. Stacked on microsoft#31154 — **review only the top four commits**; the base branch is that PR. 1. **Packed E2M1 dequantize** (`Fp4I2FConverter::decode_quad`) — decode a whole 32-bit weight word (eight codes) per step instead of one code at a time. 2. **`ORT_FP4_GEMV_DEFAULT_TILING`** — env switch to bypass the autotuner and take the default tiling, for A/B and for avoiding autotune cost in short runs. 3. **Cut memory and ALU traffic in the decode GEMV.** 4. **`kMaxProfiledExpandedRows` 8 -> 64** so MTP verify steps stay on the GEMV path. ### Motivation and Context Prior profiling established that this kernel is **ALU-pipeline bound**, not memory- or tiling-bound. On the actual Qwen3.6 decode shapes (`hidden=2048`, `inter=512`, `E=256`, `top_k=8`, bf16, SwiGLU), ncu reported for the FC1 SwiGLU-fused GEMV: > ALU 78.9%, DRAM 7.3%, occupancy 21% (register-limited) That is why the levers here are instruction-count levers. Two things were measured and explicitly **dropped** because of it: smaller `CtaN` tiling (the autotuner still picks `threads64`/`CtaN=8`; `CtaN=4` never wins because the kernel is compute-bound, not occupancy-bound), and halving scale bandwidth by storing combined scales as 1-byte e4m3 (DRAM is only ~7%, so it cannot move the needle). All numbers below: 1x H200 SXM (SM90, 132 SM, ~4.8 TB/s HBM), CUDA 13.0, Qwen3.6-35B-A3B-NVFP4 + MTP `N=3` (verify batch `M=4`). ### 1. Packed E2M1 dequantize `prmt` selects four bytes per instruction, so a 4-element magnitude lookup costs one instruction instead of four. Bit-identical to the per-element path (same magnitude tables, same sign handling). The FP4 GEMV kernel SASS shrinks ~30%, and the two QMoE GEMVs drop: | kernel | before | after | |---|---:|---:| | fc1 (SwiGLU-fused) | 33.2 µs | **26.2 µs** | | fc2 | 30.2 µs | **22.2 µs** | ### 2. Cut memory and ALU traffic — −0.46 ms/step (−5.1%) The scales of the `CtaN` columns a block owns sit `Interleave` elements apart, so for the non-interleaved ColumnMajor layout (`Interleave == 1`) the whole `CtaN`-wide scale vector is contiguous and can be fetched with one wide access instead of `CtaN` scalar ones. This matters far more than the byte count suggests: with a groupwise scale (NVFP4 `GroupSize = 16`) and `StepK = 8`, a warp's 32 lanes cover 16 distinct scale rows that are `n` elements apart, so *every* scale load touches 16 different sectors — `CtaN * 16` sectors, using 2 bytes out of each 32-byte sector. Per-kernel (graph OFF, 40 launches/step each): | kernel | before | after | |---|---:|---:| | `moe_gemv_interleaved_swiglu_kernel` | 0.956 ms/step | **0.678 ms/step** | | `moe_gemv_kernel` | 0.700 ms/step | **0.494 ms/step** | | **family total** | **1.657 ms/step** | **1.174 ms/step** | End-to-end (4 interleaved `.so`-swap reps per arm): * before: 8.973 / 9.009 / 8.992 / 9.017 * after: 8.567 / 8.508 / 8.498 / 8.583 **8.998 -> 8.539 ms/step.** No overlap between the two sets. ### 3. `kMaxProfiledExpandedRows` 8 -> 64 The fused GEMV rejects `expanded_num_rows > kMaxProfiledExpandedRows`. Qwen3.6 is top-8, so single-token decode expands to 8 rows (accepted), but an MTP verify does not: an `(N+1)`-token verify for `num_speculative_tokens = N` expands to `(N+1) * 8` rows, i.e. **up to 64 for N=7**. Those steps fell out of the window and back onto the dequantize + CUTLASS grouped-GEMM path, which re-dequantizes all 256 experts per token. The impact of that fallback is large: with the limit at 8, the 2-token verify (expanded 16) dropped MTP to **~2.4 tok/s**; raising the limit put it at **~30–55 tok/s (12–23x)**. 64 covers the `N=3` shape used today with headroom to `N=7`. ### Tests * `onnxruntime_provider_test` FP4/FP8/QMoE: 18/18 pass. * `onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py`: 22/22 pass, including new multi-token GEMV cases and a `gemv_mode="0"` dequant-fallback companion on the identical shape, so both must match the same exact dequantized reference. ### Methodology note End-to-end deltas are quoted as **ms/step** from a fixed-step measurement, never tok/s: any numerics change alters the generated sequence and therefore the MTP acceptance rate, which swamps the speed delta. Per-kernel durations are taken with CUDA graphs **off** — `nsys --cuda-graph-trace=node` inflates durations ~35% globally and up to 3.8x for large-grid kernels. > [!IMPORTANT] > `Fp4I2FConverter::convert()` gained a `PairInterleaved` template parameter in microsoft#31154. The > packed path added here assumes the **plain** nibble order (nibble `j` of the word is logical > element `j`), which is what its `prmt` selectors encode, so it is nested inside > `if constexpr (!PairInterleaved)`. Please check that guard carefully during review — applied > without it, the pair-interleaved SM80 layout would silently decode to the wrong values.
…oft#29611) ### Description Accumulate directly in output_element_t (e.g., f16 for f16 models) instead of hardcoding an f32 accumulator in the MatMulNBits wide-tile shader. **Intel Panther Lake** | | Prefill Length | Default Prefill TPS | Optimized Prefill TPS | Improvement | | :--- | ---: | ---: | ---: | ---: | | gpt-oss-20b-ONNX | 128 | 305.70 | 344.49 | 113% | | gpt-oss-20b-ONNX | 1024 | 396.50 | 429.80 | 108% | | Phi-4-mini-instruct-ONNX | 128 | 515.90 | 592.36 | 115% | | Phi-4-mini-instruct-ONNX | 1024 | 615.39 | 753.40 | 122% | [1] https://huggingface.co/onnx-community/gpt-oss-20b-ONNX [2] https://huggingface.co/onnx-community/Phi-4-mini-instruct-ONNX ### Motivation and Context See above.
…ch (microsoft#31480) ### Description When GroupQueryAttention runs with a quantized KV cache, the K cache and the V cache were dequantized by two separate kernel launches. They have identical shapes and identical per-head scale layouts, so the second launch adds a full grid setup and a second pass over the same index arithmetic for no reason. This change dequantizes both caches in a single launch. The kernel is moved into a new `group_query_attention_qdq.cuh` header and given a `2 *` grid in the cache dimension, so one block range covers K and the other covers V; the buffer pointers and scale pointers are selected from the block index. Output is bit-identical to the two-launch path -- the per-element arithmetic is unchanged, only the launch geometry differs. ### Motivation and Context This is on the decode path of speculative-decoding (MTP) workloads, where the launch is issued every layer, every step, and the per-launch fixed cost is a meaningful fraction of a short kernel. Halving the number of launches removes that fixed cost without changing numerics. Measured on H200 (SM90) with a Qwen3.6-35B-A3B MTP configuration.
Add cuda plugin ep support
Adds odd-N support to WebGPU subgroup-matrix MatMul by padding constant FP16 weights to an even row stride and caching the result.
…icrosoft#31685) ### Description <!-- Describe your changes. --> Add onnxruntime/test/providers/webgpu to plugin-ep-webgpu/paths.txt. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Missed a test path.
…ft#31683) ### Description <!-- Describe your changes. --> Announce JSEP deprecation to developers via docs in the repo. The main doc is `docs/JSEP_Deprecation.md`. Also update the JSEP to WebGPU EP migration design doc to include this initial step as well as some other clarifications. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> JSEP deprecation. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Add more paths related to cuda plugin ep to address release note skills comment microsoft#31669 (comment).
…seqlens (microsoft#29752) ## Summary - Fix TurboQuant WebGPU kernels to use `seqlen_k[batch]` instead of `seqlen_k[0]`, enabling correct quantized KV-cache handling for `batch_size > 1`. - Clamp per-batch past sequence lengths to prevent unsigned underflow for right-padded prefills. - Preserve the host dispatch and cache layouts by passing the batch-wide copy length and physical past-cache length through uniforms. - During graph capture, read the batch-global total sequence length from `total_sequence_length_input` when preparing indirect dispatch. - Select the concatenated multi-RoPE cache using the global total sequence length while retaining per-batch lengths for positioning and padding. - Apply the graph-capture total-length handling to the standard and TurboQuant packed-QKV rotary paths. - Remove the previous TurboQuant `batch_size == 1` restriction. ## Motivation The TurboQuant KV-cache copy kernels previously used `seqlen_k[0]` for every batch. Consequently, batches `1..N-1` could use the wrong past sequence length and write to incorrect cache locations. Right-padded prompts introduce another case where a batch’s logical total length can be shorter than the padded K/V input length. Direct unsigned subtraction would underflow in that case. Graph capture also requires special handling because the host-side total sequence length uniform can remain zero while the current value is supplied through a GPU input. This GPU value must be used for indirect dispatch sizing and for selecting the concatenated multi-RoPE cache bank. The multi-RoPE selection is batch-global, while rotary positions and padding checks remain per-batch. ## Test plan Regression coverage added for: - Per-batch TurboQuant decode: - `WebGPU_TurboQuant_Decode_MultiBatch_UsesPerBatchSeqlensK` - `WebGPU_TurboQuant_Decode_MultiBatch_NoRotary_UsesPerBatchSeqlensK` - Right-padded TurboQuant prefill: - `WebGPU_TurboQuant_Prefill_MultiBatch_RightPadding_NoRotary` - `WebGPU_TurboQuant_Prefill_MultiBatch_RightPadding_Rotary` - Graph-capture indirect dispatch: - `WebGPU_TurboQuant_IndirectDispatch_UsesGlobalLength_NoRotary` - `WebGPU_TurboQuant_IndirectDispatch_UsesGlobalLength_Rotary` - Concatenated multi-RoPE cache selection: - `WebGPU_IndirectDispatch_MultiRotaryCache_UsesGlobalLength` - `WebGPU_TurboQuant_IndirectDispatch_MultiRotaryCache_UsesGlobalLength` - `WebGPU_MultiRotaryCache_UsesGlobalLength_NonStaticCache` Verification: - [x] `GroupQueryAttentionTest.WebGPU_TurboQuant*`: 22 passed - [x] `git diff --check`
…crosoft.ML.OnnxRuntime.ResNet50v2Sample/SixLabors.ImageSharp-2.1.11 Bump SixLabors.ImageSharp from 2.1.9 to 2.1.11
…soft#31640) This pull request introduces improvements to the Beam Search implementation and its test coverage, specifically for FP16 (half-precision float) support. The main changes include a bug fix in the CUDA implementation and the addition of a new test to verify the output type and shape for FP16 scores. **CUDA implementation fix:** * Changed the type mapping in `LaunchBeamSearchScoreCopy` from always using `float` to using the template parameter `T`, ensuring correct type handling for FP16 and other types. (`onnxruntime/contrib_ops/cuda/transformers/generation_cuda_impl.cu`) **Test improvements:** * Added a new test, `GptBeamSearchFp16_ScoresOutputTypeAndShape`, which runs the GPT-2 Beam Search model with FP16 outputs and verifies that the output tensor has the correct type (`FLOAT16`) and expected shape. (`onnxruntime/test/contrib_ops/beam_search_test.cc`) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This pull request updates the scratch buffer allocation method in the `DeformConv` CUDA kernel implementation. The main change is a switch from using a generic allocator interface to a more specialized scratch buffer utility, which may improve performance and code clarity. **Memory allocation improvements:** * Replaced the use of `IAllocator::MakeUniquePtr<T>` and manual allocator retrieval with `GetScratchBuffer<T>` for allocating the `col_buffer` in `deform_conv.cc`. This streamlines temporary buffer allocation and ensures the buffer is tied to the compute stream. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…crosoft#31158) ### Description Adds two CUDA contrib operators that fuse the elementwise chains around the gated-delta (linear attention) block: **`com.microsoft::LinearAttentionGate`** ``` decay = decay_scale * Softplus(a + dt_bias) beta = Sigmoid(b) # optional second output ``` `dt_bias` and `decay_scale` are float32 per-head vectors of length `H`; `decay_scale` is the already-negated `-exp(A_log)` factor. **`com.microsoft::GatedRMSNorm`** ``` Y = X * rsqrt(mean(X^2) + epsilon) * scale * SiLU(gate) ``` Both are registered for `float`, `float16` and `bfloat16`. ### Motivation and Context Reference implementations compute the decay in float32 because `exp(decay)` inside the recurrence exponentially amplifies any precision loss. Exporters therefore emit `Cast -> Add -> Softplus -> Mul -> Cast` — five kernel launches over a tensor with only `num_heads` elements per token. At that size every launch is pure overhead: the work per kernel is a few hundred elements while the launch/replay cost is fixed. The gated RMSNorm chain has the same shape of problem. These ops keep the intermediates in float32 registers, so a single launch replaces each chain. The float32 intermediate is preserved exactly, so the fused output is **bit-identical** to the unfused graph and no accuracy gate is needed. ### Measured impact 1x H200 SXM (SM90, 132 SM, ~4.8 TB/s HBM), CUDA 13.0, Qwen3.6-35B-A3B-NVFP4 + MTP `N=3` (verify batch `M=4`), CUDA graph on, 200-step decode, 4 interleaved reps with the first discarded: | rep | before (ms/step) | after (ms/step) | |---|---:|---:| | 2 | 10.295 | 9.824 | | 3 | 10.331 | 9.817 | | 4 | 10.340 | 9.832 | | **mean** | **10.322** | **9.824** | **−0.50 ms/step (−4.8%).** No overlap between the two sets. Graph-level accounting confirms the mechanism is launch/replay overhead, not arithmetic: * `text.onnx` node count: **1973 -> 1583** (390 nodes removed: 30 decay chains + 30 gated norms). * In-graph time: 8.945 -> 8.604 ms/step (−0.341 ms). * The per-node graph-replay gap model predicts `0.89 us x 390 = 0.347 ms`, against a measured −0.341 ms. Output is bit-identical, and the MTP acceptance rate is unchanged in practice (495–537 vs 503–544 tokens per 200 steps). ### Tests `onnxruntime/test/contrib_ops/linear_attention_gates_op_test.cc` — fp32/fp16/bf16, with and without the optional `b` / `beta` output, checked against the unfused chain. > [!NOTE] > Stacked on microsoft#31157 (this PR's base branch) because both edit `bert_defs.cc` and > `docs/ContribOperators.md`. `docs/ContribOperators.md` still needs regenerating > (`python tools/python/gen_contrib_doc.py` after a build) — draft until then. Review only the > top commit. --------- Co-authored-by: GitHub Copilot <copilot@example.com>
### Description
Wires up the WebGPU **PagedAttention** kernel end-to-end for
continuous-batching / variable-Q-length workloads. Dispatch path:
```
→ scatter K/V into paged cache (RunScatterKVToPagedCache)
→ gather paged K/V into padded BNSH scratch (RunGatherKV)
→ unpack packed varlen Q into LEFT-aligned BSNH scratch (RunUnpackQuery)
→ ApplyFlashAttention over padded scratch
→ repack padded output back to (token_count, hidden_size) (RunRepackOutput)
```
This is Phase 1 of a planned 4-phase rollout; see the roadmap at the
bottom for the delivery plan. The v1 kernel is `MLFloat16`-only;
`softcap`, `local_window_size`, and `bfloat16` are explicitly rejected
with `NOT_IMPLEMENTED`.
### FlashAttention: optional `seqlens_q` input
The existing FA shader clamps `past_sequence_length = total_kv_b −
max_seqlen_q` to zero on underflow. That clamp is only correct for
LEFT-aligned Q with `past = 0` (the GQA
`BatchedRightPaddedRotaryPrefill` scenario). Under continuous batching,
`past_b > 0` while `q_len_b < max_seqlen_q` is common, and the clamp
silently under-counts `past_len_b` — causing real Q tokens to leak
future KV positions through the causal mask (~85% output mismatch in the
mixed-`q_len` test).
The fix adds an **optional per-batch new-Q-length** input to FA:
- `FlashAttentionProgram` and `FlashAttentionDecodeQKVProgram` gain a
`use_seqlens_q_` template-conditional gate and a `seqlens_q` shader
input.
- When bound, the shader computes `past_sequence_length_b = total_kv_b −
seqlens_q[b] = past_len_b` — always non-negative and correct for any
`(past, q_len)` combination.
- Non-PA callers (GQA / MHA / Attention) pass `nullptr`, `use_seqlens_q_
= false`, and the shader takes the byte-identical `#else` branch with
the pre-existing clamp. Zero regression risk.
- `use_seqlens_q_` is included in each program's `CacheHint` to prevent
pipeline-cache collision.
### PagedAttention: LEFT-aligned Q layout
`RunUnpackQuery` places real Q tokens at padded slots `[0, q_len_b)`
with padding at `[q_len_b, max_seqlen_q)`; `RunRepackOutput` mirrors
that layout. Matches GQA's existing convention and lets FA's
`use_seqlens_q` path compute the correct `past_len_b`.
### Test coverage
| Suite | Result | Notes |
|---|---|---|
| `TestPagedAttentionWebGpu` (Python parity) | **32 / 32** | MHA + GQA,
packed on/off, `batch_size ∈ {1, 2}`, `sequence_length ∈ {1, 4, 16}`,
`total_sequence_length ∈ {32, 64}`, `block_size = 256` |
| `WebGpuPagedAttention.EndToEnd_*` (C++) | **5 / 5** | Includes
`EndToEnd_MixedPrefillDecode_MultiBatch_VariablePast`, the exact bug-fix
path |
| `GroupQueryAttentionTest.*_WebGPU` (regression) | **31 / 31** |
Includes `BatchedRightPaddedRotaryPrefill_WebGPU` and
`BatchedRightPaddedRotaryPrefillFlashAttention_WebGPU` — unchanged since
GQA does not pass `seqlens_q` |
| MHA / Attention / TurboQuant / QKNorm WebGPU tests | **22 / 22** | No
regressions |
### Files changed
- **New:** `paged_attention_gather_kv.wgsl.template`,
`paged_attention_unpack_query.wgsl.template`,
`paged_attention_repack_output.wgsl.template`.
- **Modified:** `flash_attention.{cc,h,wgsl.template}`,
`flash_attention_decode_qkv.wgsl.template`, `paged_attention.{cc,h}`,
C++ + Python tests, design doc.
- **Renamed:** `test_paged_attention_cuda.py → test_paged_attention.py`
(adds `TestPagedAttentionWebGpu`).
### Roadmap
Work will be split into 4 phases and delivered incrementally as time
permits (not back-to-back).
- **Phase 1 (this PR).** Initial functionally-correct implementation
using the gather-then-flash fallback path described above. Goal is early
review and unblocking downstream work.
- **Phase 2.** Optimized implementation: KV-page-aware prefill and
decode FlashAttention kernels that back the PagedAttention op directly,
with the Phase 1 gather-then-flash path retained as a fallback for cases
FlashAttention doesn't cover. This also lifts `softcap` and
`local_window_size` inside `FlashAttentionProgram`, which also drops
GQA's `CanApplyFlashAttention` bailouts.
- **Phase 3.** Support the features introduced by
microsoft#29912 — quantized KV cache (`T_CACHE`), head-sink,
QK-Norm — starting with quantized KV.
- **Phase 4.** Tuning driven by real-world model traces.
Not planned for this PR: `T = bfloat16` (blocked on Dawn stability),
graph-capture with `attention_metadata` sizing bound (design-doc §4.4),
MLA / LATENT layout (Phase 4+ as customer need materializes).
CI coverage note: `TestPagedAttentionWebGpu` currently runs on **zero**
CI legs — the two WebGPU CI workflows are build-only, and
`nightly_webgpu.yml` / macos-ci run `--test` without
`--enable_transformers_tool_test`. The `WebGpuPagedAttention.EndToEnd_*`
C++ gtests DO run on `nightly_webgpu` and macos-ci. Wiring the Python
parity suite into a WebGPU CI leg is a small follow-up.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary Update the CUDA plugin pipeline to publish release-ready Linux archives and split NuGet packages by canonical .NET RID. ## Key changes - Publish Linux `.tar.gz` archives in the `cuda_ep_cuda12_linux_gz` / `cuda_ep_cuda13_linux_gz` artifacts alongside the existing platform zip artifact. - Generate one NuGet package per enabled RID: - `Microsoft.ML.OnnxRuntime.EP.Cuda12/13.win-x64` - `Microsoft.ML.OnnxRuntime.EP.Cuda12/13.win-arm64` - `Microsoft.ML.OnnxRuntime.EP.Cuda12/13.linux-x64` - `Microsoft.ML.OnnxRuntime.EP.Cuda12/13.linux-arm64` - Keep a shared `.csproj`; `pack_nuget.py` selects the package ID and exact native RID contents at pack time. - Publish all RID-specific package IDs in pipeline metadata and update the Windows GPU NuGet test to consume the `win-x64` package. - Update packaging documentation and local examples to use the canonical RID names. ## Validation - YAML and project XML parsing passed. - Ruff check and format check passed for `pack_nuget.py`. - Editor diagnostics reported no errors in touched files. - End-to-end dry packing produced four packages, each containing only its matching runtime directory. - PowerShell/tar archive construction was tested locally. Azure pipeline execution was not run locally.
…9789 (microsoft#29826) Add regression test for microsoft#29789
## Description Stacked on microsoft#31159; review only the top commit. This removes the standalone FP4 QMoE fc1 activation expansion during GEMV decode. Instead, fc1 maps each permuted row back to its source token with `permuted_row_to_source_row[row] % num_rows`; fc2 remains unchanged because it consumes the expanded fc1 output. ## Summary of Changes - Pass the permuted-row-to-source-row mapping through all FP4 interleaved SwiGLU GEMV launch variants. - Read fc1 activations directly from the original input while preserving PR 31159's SM80 pair-interleaved weight layout. - Keep the legacy path available with `ORT_DISABLE_FP4_GEMV_SKIP_EXPAND=1` for same-binary comparison. - Add an exact-output parity test for the MTP shape (`num_tokens=3`, `top_k=8`) with skip-expand enabled and disabled. ## Performance H200, Qwen3.6 35B A3B NVFP4, MTP N=3, paired same-binary A/B: - Removes 40 `expandInputRows` launches per decoding step. - Removes 94.651 us/step of activation expansion. - Adds 10.377 us/step to fc1 source-row lookup. - Saves 84.274 us/step across named kernels. - Saves 0.097695 ms/step median end-to-end (1.314%). ## Testing - `lintrunner -a` on the five changed files. - Compiled `moe_gemv_fp4.cu` and `moe_quantization.cc` against the exact microsoft#31159 head. - New expanded-vs-skip-expand regression passes with exact tensor equality. - 23 NVFP4 QMoE CUDA tests pass locally; the separately isolated large-input scaling test fails identically with skip-expand enabled and disabled on the pre-existing integration binary. ## Checklist - [x] Tests added/updated - [x] No breaking changes - [x] No documentation changes required
### Description Adds int32 and uint32 data type support to the Cumsum operator in the WebGPU execution provider. The op is used by [Phi-4-mini](https://huggingface.co/webnn/Phi-4-mini-instruct-onnx-webnn/blob/main/onnx/model.onnx) (GQA decomposition). CumSum's WebGPU kernel only advertised float types (float16/float32), so int32/uint32 inputs silently fell back to the CPU EP. Widen the T type constraint to WebGpuSupportedNumberTypes() (adds int32/uint32). ### Motivation and Context No shader or C++ dispatch change is needed. The kernel is not templated and the shader is written entirely against the generic value-type alias (output_value_t): the accumulator is `var sum : output_value_t = 0` and the reduction is a plain `sum = sum + ...`. WGSL resolves the `0` literal and `+` for i32/u32 exactly as for the float case (integer add wraps, matching the CPU CumSum reference), and the loop bounds/indices are already i32/u32 and independent of the element type. So enabling the two integer types is purely a matter of relaxing the type-constraint whitelist.
…t#31665) ### Description <!-- Describe your changes. --> Adds a bounds check to the DML EP's `OnnxTensorWrapper` so that a `TensorProto` whose declared shape implies more data than its backing buffer actually holds is rejected at construction, instead of being handed to operator kernels. - New helper `VerifyTensorProtoFitsInBuffer` in `MLOperatorAuthorImpl.cpp` computes the byte size implied by the tensor's dims and element type, and fails with `E_INVALIDARG` when the buffer is smaller - Called at the end of the `OnnxTensorWrapper` constructor, covering the `raw_data`, typed-field, and external-data paths - Adds `onnxruntime/test/providers/dml_onnx_tensor_wrapper_test.cc` with positive / negative cases. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> DML reliability improvement. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…#31606) ## Summary The ARM64 symmetric quantized GEMM tests (`MlasSymmQgemmTest`) only ever feed A and B with `MatrixGuardBuffer::GetBuffer()`'s default fill, which is confined to `[21, 64)` — even `ExecuteLong`'s large M/N/K sweep never touches an int8 extreme. This adds `SymmQgemmS8SignedInputTest`, mirroring the `QgemmS8U8SignedInputTest` pattern from microsoft#29787, with A and B both explicitly filled with int8 extremes (`-128, -1, 0, 1, 127, ...`) across a K/M/N/offa grid sized around the kernel's actual block structure (`PackedK=16`, `StrideM=4`, N aligned to 16). This is a test-only change; nothing under `onnxruntime/core/mlas/lib/` is touched. While adding these tests, I found a correctness bug in the plain-NEON (non-dotprod) `MlasSymQgemmS8KernelNeon` kernel and filed it separately as microsoft#31573. The new tests here pass because they exercise the SDOT dispatch (default on both machines I verified this on); the NEON dispatch only gets exercised on ARM64 cores without dot-product support. ## Testing `onnxruntime_mlas_test`, full suite, no regressions: - Apple M1 (macOS): 29227/29227 passed - Neoverse-N1 (Oracle Cloud A1, Ubuntu): 36020/36020 passed `*SymmQGemmS8_Int32_SignedInput*`: 1248/1248 passed on both (624 cases x SingleThread/Threaded).
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 8, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.