[zibi-pplx]

Summary

Build and register CH seed and derived templates through the existing services, preserving exact environment, runtime and guest identity. Derive installed guest-seed versions from both executables, reject disagreements, and flush and stop the guest before publishing its disk. CH finalization powers off because a reboot keeps its VMM resident. Run guest-version probes asynchronously with a five-second deadline and kill-on-drop so a broken executable cannot block the builder indefinitely.

Resolve base-snapshot and in-node pool requests from the registered template's runtime instead of hardcoding Firecracker. Pin the selected build hash when creating a pool member. Request-construction coverage iterates all runtime kinds; environment qualification and lifecycle admission remain responsible for whether snapshots are supported.

Share the architecture-to-boot-format contract, validate bundled recipes without a runtime-directory allowlist, and document the cold-only and snapshot-capable environments. Existing templates are not relabeled to gain capabilities.

Validation

The two builder scratch/version tests and affected-package Linux all-feature Clippy passed after making version probes asynchronous. Complete live base-snapshot/two-member-pool scenarios passed for CH and environment-bound Firecracker, including guest markers and cleanup.

The all-runtime base-snapshot request-construction test passed on Linux, including gVisor identity forwarding without claiming gVisor snapshot support. Related Linux clone/admission tests passed. The prior review pass passed affected-package Clippy and 52 targeted Linux tests.

The current x86-64 working build created a CH base snapshot and started a usable guest from it without cold fallback. After preserving disk-level clone metadata, a complete live scenario passed two-member pool fill, adoption, resume, guest marker verification, deletion, lease cleanup and policy restoration. Earlier combined-stack qualification built CH seed/derived templates and exercised persisted memory and S3 materialization. These template/pool runs do not qualify cluster placement or encryption; the prerequisite runtime PR records the separate encrypted lifecycle and S3 materialization evidence.

Severity and risk

Opt-in template support; fleet defaults do not change. Workers and builders must share allowed environments and authenticated artifacts. The pool runtime selection change also affects Firecracker and pins the build selected by template-manager. Unsupported or mismatched bindings refuse rather than silently select another runtime.

Rollout

Deploy the complete CPU sequence #925 → #926 → #928 → #929 → #930 → #931 → #960 before enabling CH. Keep compatible services and boot artifacts while CH snapshots exist. ARM64 execution, GPU, Windows and cross-node placement are not qualified here. Encrypted deployments additionally require the runtime PR's encrypted-volume prerequisites and a qualified credential authority.

AI assistance: Codex drafted code, tests and this description.

[the-gigi-pplx]

Review — 6 findings

Medium

1. crates/libs/build/mgr/src/lib.rs:1095 — Removing the requires_durable_stop() branch makes a failed Stop fatal for every build, including plain Firecracker recipe builds.
   Before, a non-runtime-bearing Firecracker build that hit a Stop error logged a warning and copied the rootfs anyway. Now it deletes the sandbox and fails the build. The description states the new rule ("require acknowledged Stop before copying the rootfs"), and it's the safer extreme: a copy from an unconfirmed stop can publish an inconsistent image. The cost on the other side is that any transient Stop failure (cleanup permit, node-manager RPC, fence conflict) now fails builds on the existing Firecracker path that used to succeed. Nothing tests either branch. Please add a test for the Firecracker Stop-failure path. If transient Stop errors turn out to be common on builders, consider one bounded retry of Stop before failing.
2. crates/libs/build/mgr/src/lib.rs:1755 — The new /space/data/spaced fallback can bake guest binaries whose version doesn't match the stamped metadata.
   detect_spaced_version() (lib.rs:1759) still takes the version from the builder's workspace Cargo.toml or CARGO_PKG_VERSION. But the binaries now come from whatever the node has installed, which can be another release mid-rollout. The seed then records a spaced_version / spaced_compatibility_family its guest binaries don't have. Before, this deployment shape failed loudly with "could not locate spaced/spaced-svc". Either read the version from the binaries that are actually copied, or keep the fallback out of scratch and OCI builds.

Low

3. crates/libs/build/mgr/src/vm.rs:25 — The CH exit command (sleep 1; exec poweroff -f) has no fallback and no sync of its own. The Firecracker command syncs and falls back to reboot -f. If the preceding GUEST_FLUSH_CMD fails (only logged at lib.rs:1065) or poweroff is missing in a derived image, the build waits out SHUTDOWN_TIMEOUT and Stop kills a live VM. The runtime-to-command mapping has no unit test.
4. crates/libs/build/mgr/src/lib.rs:69 — The rationale on GUEST_FLUSH_CMD was cut to two lines. The deleted text explained the sysrq-versus-writeback race and the e2fsck journal-recovery loss of the final recipe step, which is the reason the blocking flush exists, and the comment at lib.rs:1062 still says "see GUEST_FLUSH_CMD" for that reason.
5. templates/firecracker/seed/build.sh:253 — The standalone build.sh forwards SPACE_SEED_RUNTIME but not SPACE_SEED_STATE_FORMAT / SPACE_SEED_KERNEL_FORMAT, and always defaults a Firecracker state-format version. So SPACE_SEED_RUNTIME=cloud-hypervisor ./build.sh reaches the new "state format name and version must be supplied together" abort in rootfs-builder.sh. Either forward the new variables or refuse non-Firecracker runtimes there explicitly.
6. crates/libs/build/mgr/src/recipe/tests.rs:1237 — bundled_recipe_paths hard-codes ["firecracker", "cloud-hypervisor"]. A future templates/<runtime>/recipes directory escapes the bundled-recipe guards silently. Globbing templates/*/recipes/*.toml removes the list.

Verified against 4de638f92c07. The current head ca5e6f59fb21 is the same patch restacked on PR #930's new head (git range-diff shows it unchanged). Read the diff against PR #930's head. The Firecracker seed path through build-seed.sh / rootfs-builder.sh still requires a state-format version and defaults the format name and vmlinux kernel format, so that path is unchanged. I didn't run the build-manager tests or a seed build.

[the-gigi-pplx]

Approved. 6 non-blocking findings in the review comment above.

[zibi-pplx]

@the-gigi-pplx Reviewed these against the complete seven-PR CPU sequence. The accepted fixes are applied and absorbed locally, not pushed at the author’s request; the GitHub heads do not contain them yet. The later GPU/Windows drafts were left untouched.

• Installed guest-seed fallback metadata now comes from both selected executables’ --version; disagreement refuses the build. A regression covers matching and mismatched pairs. Packaged workspace assets retain their existing version source.
• CH finalization now syncs before delayed power-off. It intentionally has no reboot fallback, since reboot does not terminate CH. The exit-command test checks this choice.
• Restored the writeback/sysrq/e2fsck rationale, explicitly refused non-Firecracker use of the standalone Firecracker seed script, and made bundled-recipe checks discover runtime directories.

I retained acknowledged Stop as the publication barrier for Firecracker too: copying after an uncertain stop can publish an inconsistent image. I did not add an optimistic fallback or speculative retry. A full Firecracker Stop-RPC fault-injection build test remains a validation gap; the targeted tests do not prove that end-to-end case.

The version, exit-command and bundled-recipe Linux tests passed. Earlier live template-build evidence predates these fixes.