docs(ci): document the three-stage validation ladder (ROCMAI-431)

[juhovainio]

Depends on #463

In plain English

Documentation only, no code changes. Explains the three levels of automated testing we now have (a quick check on every pull request, a deeper check every night, and a full regression check before a release), so it is clear what each one covers and how the in-progress PRs above fit together. Needs #463 merged first so the doc matches the lane changes that PR makes.

Summary

Adds the missing framing to docs/ci-hardware-testing.md:

• A three-stage validation ladder section naming and separating the per-PR smoke gate (e2e-selfhosted.yml), the nightly coverage gate (nightly.yml, now covering 6 platforms incl. R9700/rad3 and MI350P, ROCMAI-125/ROCMAI-429), and the release-candidate regression gate (e2e-selfhosted.yml on release/**, ROCMAI-120/EAI-8761). States plainly that the release-candidate rung's trigger ships in PR ci(e2e): auto-trigger self-hosted E2E matrix on release-branch push (EAI-8761) #415 and its SDK-version pin flags in PR feat(e2e-prewarm): pin the SDK version instead of always tracking latest (ROCMAI-430) #464, both stacked but not yet merged, and that no matrix-level sdk_version: [current, n-1, n-2] axis exists anywhere (deferred in feat(e2e-prewarm): pin the SDK version instead of always tracking latest (ROCMAI-430) #464, no repo mechanism maps n-1/n-2 to a concrete version).
• A short note on xtask/src/workflow_contract.rs's self_hosted_workflow_owns_the_gpu_lanes pinning the three per-PR job names by name (EAI-7548 rationale).
• A one-sentence note on the consolidated report's (platform_slug, channel) column keying (ROCMAI-429).
• An engine is not an independently selectable axis section, quoting effective_serve_engine() from tests/e2e-cucumber/src/capability.rs verbatim, explaining why every Strix lane is lemonade-only and e2e-gpu (MI300X) is the only per-PR vLLM lane.
• A short addendum to "The shared pre-warmed runtime" mentioning e2e-prewarm's --version/--build-date pin flags (ROCMAI-430).

No lane, artifact, or job-order changes in this PR, so every contract-tested table/list/sentence in the doc (hardware_testing_docs_cover_all_self_hosted_platforms) is left byte-identical — verified by running the full xtask suite before and after.

Every factual claim above was cross-checked directly against the workflow YAML and Rust source (not against the Jira tickets), including a correction of a prior assumption: the doc's existing canonical-artifact-list paragraph was suspected stale but is in fact still correct, since nightly.yml still uploads e2e-gpu-rad3-*-report/e2e-gpu-mi350p-*-report even though the per-PR jobs were removed.

Verification

• cargo fmt --all -- --check clean.
• cargo test -p xtask --all-targets: 155/0/0/0, including hardware_testing_docs_cover_all_self_hosted_platforms.
• grep -rn 'e2e-gpu|strix|mi350p|rad3' docs/ README.md outside this file: no hits (no duplicated lane claims elsewhere).
• Leak scan on the diff: no hits (one "private" match is a Rust-visibility reference, not sensitive data).

No Gherkin scenario: this PR only edits documentation prose, no product or test behavior changes.

[tomastola]

What the change does. Docs-only: adds a "three-stage validation ladder" section to docs/ci-hardware-testing.md (per-PR, nightly, release-candidate gates), a section on why the serve engine is not a matrix axis, and notes on the (platform, channel) report columns and the e2e-prewarm pin flags. I checked the new claims against the code at this head. The workflow triggers, lane names, nightly cron and channel matrices, effective_serve_engine behaviour and report column keying are accurate. Two claims are not (below).

Coverage. Single full read of the PR's own diff (one file), checked against the workflows, xtask and capability.rs at head 930f816. This head is a rebase of the previously reviewed 46f87e6; the PR's own diff is byte-identical, and the parent commits that moved touch none of the code this doc describes. I did not read the earlier discussion before forming the findings and did not run any tests.

Needs fixing before merge

• The e2e-prewarm paragraph (inline) describes --version/--build-date in the present tense, but that code is not in this tree, and the ladder section of the same doc says it ships in #464. Someone following the prewarm section would pass a flag that does not exist.

Non-blocking

• The per-PR row says it covers the 4 platform jobs, but e2e-gpu-strix-ubuntu already runs a [release, nightly] channel matrix per PR (inline).
• "cannot be silently undone" overstates what self_hosted_workflow_owns_the_gpu_lanes checks (inline).

Open decisions for the author (not defects)

• The effective_serve_engine source is pasted verbatim into the doc. It reads well, but it drops the source's WSL comment and will drift silently when capability.rs changes. Referencing the function by name, as the rest of the doc does, would age better. Deliberate?
• The paragraph hard-codes #415 and #464 and "as of this writing". It goes stale once either merges. A ticket reference, or a marker whoever merges them will find, is easier to keep true.

The merge decision is yours; this review is posted as a comment and files no approval or change request.

[tomastola]

The flags described here do not exist yet. This paragraph says e2e-prewarm "also accepts mutually exclusive --version/--build-date flags" and that "today no caller in this repo passes either flag yet". There is no --version/--build-date handling under xtask/src at this head, and the ladder section of this same doc (lines 43-49) says they ship in #464, which is open and stacked on the also-open #415. The code and the ladder paragraph are the accurate side; this paragraph is not. Reword it as future/conditional ("#464 adds…"), drop it in favour of the single mention in the ladder section, or land this PR after #464.

[juhovainio]

Fixed in 6534052 — reworded to future/conditional tense and cross-referenced to #464 (stacked on #415), matching the ladder section. No longer reads as already landed.

[tomastola]

Per-PR row is incomplete (non-blocking). "The 4 jobs in the Platforms table" leaves out that e2e-gpu-strix-ubuntu in e2e-selfhosted.yml already has strategy.matrix.channel: [release, nightly] and uploads e2e-gpu-strix-ubuntu-${{ matrix.channel }}-report (from the ROCMAI-125 parent commit). So the per-PR gate is really 5 job runs and is not release-channel-only; the channel matrix and the two-column report (line 177) are not nightly-only. Worth stating in the row.

[juhovainio]

Fixed in 6534052 — the row now states e2e-gpu-strix-ubuntu runs both its [release, nightly] channel legs per PR (ROCMAI-125).

[tomastola]

"cannot be silently undone" overstates the test (non-blocking). self_hosted_workflow_owns_the_gpu_lanes (xtask/src/workflow_contract.rs) only asserts sh.contains(job) for the three job keys, a substring check that any mention in a comment satisfies. It also does not cover e2e-wsl, which a separate test handles. Suggest "requires the three job keys to appear in the file".

[juhovainio]

Fixed in 6534052 — replaced with your suggested wording: the test requires the three job keys to appear in e2e-selfhosted.yml, so moving one back to ci.yml fails CI rather than merging unnoticed.

[juhovainio]

Addressed the review in a01b7b8d (also rebased onto #463's latest, which landed its own review fixes):

• e2e-prewarm tense: the --version/--build-date paragraph claimed present-tense availability, contradicting the ladder section two paragraphs up which correctly says that flag pair ships in feat(e2e-prewarm): pin the SDK version instead of always tracking latest (ROCMAI-430) #464. Reworded to future tense ("will also accept"), consistent throughout.
• Per-PR row: now notes e2e-gpu-strix-ubuntu runs both its [release, nightly] channel legs per PR, not just "4 flat jobs."
• "cannot be silently undone": reworded to match what self_hosted_workflow_owns_the_gpu_lanes actually checks — the three job keys appear as text in e2e-selfhosted.yml, not a full structural pin.

Left the two open-decisions-for-the-author items (the pasted effective_serve_engine source, the hardcoded #415/#464 references) untouched — non-defects per your review, and I don't have a strong opinion either way.

cargo fmt --all -- --check and cargo test -p xtask --all-targets (155 passed, including hardware_testing_docs_cover_all_self_hosted_platforms) both clean.