[zibi-pplx]

Summary

Connect Cloud Hypervisor to fenced node launch, control and sandbox lifecycle while preserving Firecracker/E0 defaults. Authenticate opened executable and kernel descriptors, attest control sockets to the registered process lifetime, and require containment and Landlock. Prefer fs-verity; unsupported filesystems use catalog SHA-256 verification of protected source inodes.

Add pause/resume and persisted memory capture/restore through a separately qualified state environment. Preserve guest device identities while rebinding host resources. Reuse existing publication and S3 materialization, including replacement of retained instances through node-manager's fenced operation. Keep historical cold-only environment identities unchanged.

Preserve execution metadata in donor clones in the sandbox record, beside the checkpoint and at the disk root. Parse disk metadata, verify its donor identity and rebind only the sandbox ID before publication. Refuse restore when the checkpoint runtime or guest contradicts the saved sandbox binding. Permit local base-snapshot and pool creation for memory-state-qualified template bindings; explicit environment overrides remain cold-only and cluster creation remains gated.

Require fresh, complete execution inventory and matching guest facts for non-Firecracker Direct placement. Keep missing-inventory and lazy-kernel compatibility fallbacks limited to Firecracker; base-snapshot and restore candidates must report the matching loadable state contract. These approved query changes add no schema migration and do not enable cluster CH admission.

Keep lifecycle identity reads outside the diagnostic semaphore so saturated diagnostics cannot reject launch or snapshots. Admit idle/paused restore retries, roll back a loaded runtime after finalization failure, and apply the admitted disk limiter. Use operation-specific control deadlines, scaling state copies by memory size, and report timeouts as uncertain outcomes. Pool admission checks both runtime capability and the architecture-specific environment qualification.

Bound endpoint probes and blocking work. Reconcile exited Started executions for both runtimes using positive process-exit evidence and fenced cleanup, preserving disks. Inconclusive ownership never releases resources.

Encrypted capture/restore uses the existing volume and key authority contracts. Bind the encrypted workspace into the contained VMM, keep provider state and RAM staging on that volume, and stop/drain before sealing. Persist typed guest-memory allocation in existing JSON metadata so restore can reserve capacity without a plaintext RAM file; decrypted state must match it. Older plaintext metadata retains file-size fallback. No SQL or protobuf change.

Nested snapshot cleanup pins the mount parent before inspecting or unmounting entries. Planted parent symlinks are left for unlinking, so nested encrypted mounts cannot follow a substituted parent or stall forever on a loop.

Validation

Rebase validation: 66 targeted Linux tests, affected-package all-feature library/test Clippy, and workspace formatting passed. Main's runtime-upgrade fences and environment-partition cache behavior are preserved. Template-manager now restricts automatic pool refill to legacy-partition builds, so the earlier CH pool runs below do not qualify pools at this rebased revision. No live-node test was repeated after rebase.

CI regression: the existing cleanup fixture assumed every mount entry was a direct child. The new nested encrypted mount exposed that assumption; after repairing the fixture, it also reproduced a parent-symlink-loop cleanup failure. Both failures reproduced on Linux before the fix. All 14 unprivileged cleanup-helper tests and the private-namespace stacked-bind test passed afterward, including direct and nested snapshot mounts; node-manager all-feature library/test Clippy passed with warnings denied.

Encryption revision: 13 targeted Linux metadata/allocation tests passed, including legacy shape, invalid allocation, artifact identity and restore size mismatch. Seven runtime-control tests cover fixed encrypted/plain paths and deadline behavior. Affected common/execution/node/sandbox Linux all-feature library/test Clippy passed with warnings denied, workspace formatting passed, and architecture checks passed. The deployed x86-64 build passed two encrypted suspend/restore cycles, including manager restart, RAM/disk/boot-identity preservation, sealed-only publication, and sandbox/lease cleanup. The key authority was a disposable one-sandbox fixture. Encrypted S3 upload/download and restore also preserved RAM/disk/boot identity after removal of the local checkpoint. That smoke ended with an SSH timeout in its final lease inspection; independent verification found no sandbox, lease or crypt device remaining. Local encryption is a complete smoke pass; the S3 run is recorded as behavioral success plus an infrastructure-error verdict. Temporary key/storage fixtures and the test bucket were removed. These runs do not qualify production credential authorities.

Round-five fixes: 28 targeted Linux tests passed across control/boot codecs, clone metadata, diagnostic saturation, lifecycle admission and builder version checks. Affected-package Linux all-feature library/test Clippy passed with warnings denied. The CI cache-fixture failure reproduced under a non-root Linux user; correcting its directory depth made the fallback test pass without changing production checks. Ten non-privileged cache tests passed; two fs-verity tests require the privileged filesystem lane.

The checkpoint clone regression failed before the fix and passed afterward in Linux. All six clone tests, including the separate-UID containment test and disk-identity rebinding, passed in a disposable Linux container; node-manager Linux Clippy passed with warnings denied. Targeted Linux tests cover clone path safety, runtime/environment admission, request identity preservation and runtime control. All 96 Direct placement tests passed, including the positive-inventory matrix; the final matrix rerun and API all-features Clippy also passed. The prior review pass also passed 52 Linux tests, affected-package Clippy, formatting, and smoke syntax checks.

An x86-64 working build containing these changes passed CH pause/resume, full snapshot, two suspend/restore cycles and manager restart recovery with RAM, disk and boot identity preserved. Cluster qualification remains in progress. Earlier combined-stack runs passed the corresponding Firecracker lifecycle and real S3 upload/download with a missing local checkpoint; this does not establish cluster placement or credential-broker qualification.

Severity and risk

Opt-in privileged runtime integration. Process-exit recovery and retained-instance replacement also affect Firecracker. CH restore copies RAM, so large-memory latency needs workload-specific measurement. Cluster routes remain gated. Encrypted state requires cryptsetup and a persistent credential authority; local restore passed with a disposable key authority; encrypted S3 restore passed the behavioral checks described below. Complete live x86-64 scenarios passed base-snapshot creation/start, filling two-member pools, adoption/resume, guest markers, deletion, lease cleanup and template-policy restoration for both CH and environment-bound Firecracker. These pool runs preceded the round-five restore hardening. The adapter now advertises qualified in-node warm-pool support. All 19 execution-spec tests and its all-features Clippy passed. Full end-to-end cluster recovery remains outstanding.

Rollout

Deploy #925 → #926 → #928 → #929 → #930 → #931 → #960 together before enabling CH. Install the supervisor containing #930 before enabling CH in node-manager. Keep compatible services and boot artifacts while snapshots exist; drain CH executions before rolling back to a build without CH. ARM64 execution, GPU, Windows and cross-node placement are outside this revision's qualification. Qualify the deployment's real credential authority before production encryption; retain compatible services and key authority while encrypted checkpoints exist.

AI assistance: Codex drafted code, tests and this description.

[the-gigi-pplx]

Review — 8 findings

Medium

1. crates/libs/sbx/mgr/store/src/db/sandboxes.rs:1321 — Adding Started to the periodic-recovery states lets healthy Started rows fill the 8-candidate cap, and this applies to Firecracker nodes too.
   The batch stops at 8 candidates (:1335) within a 128-rowid window (:1292-1295), the cursor advances per examined row, and the tick is 60 s (recovery.rs:2433). Nothing filters Started rows before the candidate is counted; the node_execution_id and pid checks happen later, in admit_exited_runtime. Every row carries a node_execution_id (sandboxes.rs:246, with legacy rows adopted at startup), so the is_none() guard at cleanup.rs:796 doesn't exclude Firecracker rows. On a node with mostly running sandboxes, the cursor covers about 8 rows per tick instead of up to 128. That makes Paused/Resuming/Checkpointing/Suspending recovery, and runtime-exit detection itself, take up to 16x longer to come around. The store test moved its filler rows from Started to Stopped (db/tests.rs:178), so the Started-dense case is no longer covered. Consider a separate cursor or budget for runtime-exit probing, and keep a Started-dense case in the test. Also, the comment at startup_resident.rs:88-89 ("Historical runtimes without that fence stay untouched") doesn't describe the fleet as it is, for the same reason.
2. crates/libs/node/runtime/core/src/runtime_inventory/managed.rs:477 — Node-manager and the supervisor decide the SHA-256 fallback on different files.
   Node-manager picks CatalogSha256 because stage_release_at finds no verity support on the execution-cache parent (artifact_store/execution_cache.rs:154-166), and then publishes source.path(), the store or baked path, which that file's own comment (:152-153) says can sit on another filesystem. The supervisor's AuthenticatedFile::open_catalog measures that source path (authenticated_source.rs:76-79). Only Unsupported maps to the fallback (:116); ENODATA maps to Unsealed (verity.rs:248), which is a refusal. With a cache lacking verity and a verity-capable source filesystem holding unsealed files, every CH launch is refused. That's CH-only and fails closed, but it's a real deployment shape. Make both sides decide on the same file, without letting the request downgrade the check.
3. crates/libs/sbx/mgr/server/src/service/cleanup.rs:781 — admit_exited_runtime try-acquires the sandbox cleanup fence before it knows the runtime exited. It holds the fence across the writer lease, the row re-read and the pidfd probe (:792-805), for every Started row the cursor reaches, including healthy Firecracker ones. A concurrent Suspend gets Unavailable "sandbox cleanup is active; retry suspend" (grpc_lifecycle/suspend.rs:264-269), and Export gets Aborted "cleanup is in progress" (grpc_sandbox_transfer.rs:367-368). The window is short and the refusal is retryable, but it's a spurious user-visible error with no cleanup behind it. Probe the pinned pidfd first without the fence, and take the fence (re-validating state, execution and pid) only once exit is observed. The same block is an inline copy of try_cleanup_fence (:1639-1646); call the helper.

Low

4. crates/libs/sbx/mgr/server/src/service/cleanup.rs:777 — The pidfd probe takes a cleanup_io permit (4 permits, cleanup_support.rs:458) shared with real cleanups. During delete storms each tick queues behind cleanup I/O and holds slots a delete could use. A pidfd poll isn't storage I/O and doesn't need the semaphore.
5. crates/libs/node/mgr/server/src/service/orphan_inventory.rs:240 — "space-netgtw-svc" is 16 characters, and /proc/<pid>/comm is capped at 15, so this arm never matches. I confirmed it on Linux: a copy of sleep named space-netgtw-svc reports comm space-netgtw-sv. Gateways are still found through the new exe-readlink fallback, but every ordinary host process now pays a readlink, and any non-NotFound error marks the page incomplete (:246, :89-93). Restore space-netgtw-sv.
6. crates/libs/node/mgr/server/src/service/cleanup_inspect.rs:788 — recorded_execution_blocking(...)? makes InspectSandbox, and through it sbx-mgr Stats (:716-717), fail outright when metadata.json can't be decoded or doesn't match. Inspection previously didn't read metadata. Cleanup already probes both adapter roots by sandbox identity; inspection could do the same, or fall back to the Firecracker root and report the metadata error as a field.
7. crates/libs/node/mgr/server/src/service/endpoint_attestation.rs:156 — Several Firecracker hot paths now re-read metadata.json to learn the runtime kind: pause/resume (snapshots.rs:274-281), endpoint-proof capture and validation, the disk-I/O ceiling (disk_io_ceiling.rs:500-508) and sidecar launch (runtime.rs:529-541). A corrupt file now fails Firecracker pause and attestation that used to work regardless. Consider carrying the runtime kind on the slot lease, which these paths already read.
8. crates/libs/sbx/mgr/server/src/service/startup_resident.rs:111 — On non-Linux, has_exited is always Unsupported (launch/os/other.rs:110-124), so every Started row logs "runtime exit recovery deferred" on every pass and takes the fence for nothing. That's local-dev noise only. Skip the probe when pidfds aren't available.

The description already calls out that the Started-execution scan affects Firecracker and asks for Firecracker crash recovery in the canary. Findings 1 and 3 are the parts of that cost I'd want priced before the canary, since they apply to every healthy Firecracker sandbox, not just crashed ones.

Verified against 1340082f5cfa, and re-checked at aac0544d5763: the only change since is the space-nodemgr lease-selector dedup in crates/apps/node/mgr/cli, which doesn't touch any file above and which I read. PRs #931 and #960 touch none of these files. Checked by tracing each path at this head. The only thing I ran was the /proc comm-length probe for 5 on a Linux host. I didn't run the cargo tests.

[the-gigi-pplx]

Approved. 8 non-blocking findings in the review comment above.

[zibi-pplx]

@the-gigi-pplx Reviewed these against the complete seven-PR CPU sequence. The accepted fixes are applied and absorbed locally, not pushed at the author’s request; the GitHub heads do not contain them yet. The later GPU/Windows drafts were left untouched.

• Split interrupted-operation recovery and Started-process discovery into independent bounded cursors. A Started-dense regression preserves interrupted-operation progress; the process scan can examine the existing 128-row window rather than consuming the eight-operation budget.
• Check the pinned process before taking the cleanup fence. After positive exit, acquire the fence/writer lease, re-read state/execution/PID and probe again before durable cleanup. PID probes no longer consume cleanup-I/O permits; the regression runs with those permits saturated.
• For SHA fallback, publish a fresh authenticated copy into the protected execution-cache filesystem. Inventory and the launcher now authenticate the same backend. A verity-capable backend still refuses this fallback, and the request cannot select a weaker mode. Tests cover independent copied bytes, refusal on a capable backend, and corrupt cached bytes.
• Restore the truncated gateway comm name, make diagnostic inspection examine both runtime roots, and skip unsupported process-exit probing on non-Linux.

I kept fail-closed metadata checks for mutations. Adding runtime kind to a durable slot lease is a separate storage-contract change, not a safe cache-only shortcut. Diagnostic inspection is now independent of that metadata failure.

The combined local revision passed 52 targeted Linux tests, affected-package Clippy and formatting. No SQL statements or protobufs changed. Existing live evidence predates these review fixes; no new live qualification is claimed.