[the-gigi-pplx]

Summary

Restores of a full snapshot can now be kept off hosts whose CPU can't load the snapshot's vCPU state. A snapshot captured on one CPU model can fail Firecracker's PUT /snapshot/load on another (Failed to set all KVM MSRs), and a restore placed on such a host fails on every retry. Everything here sits behind SPACE_SANDBOX_RESTORE_HOST_CPU_PLACEMENT_ENABLED, which defaults to off. With it off, placement is exactly as on main.

This is the Kubernetes layer. The capture CPU recorded on suspend snapshots and the space.pplx.net/host-cpu node label come from PR #203.

What changed

With the gate on:

• Placement term. A restore pod from a remote full snapshot whose capture CPU is known gets a required space.pplx.net/host-cpu NotIn [...] term listing every published CPU that can't load it. The term goes into every required node-selector term, so it can't be escaped through the source-node shortlist. NotIn leaves unlabeled nodes eligible.
• When the term is rendered. The API adds the term only when the node cache shows at least one node the pod could land on whose CPU can load the snapshot. That check accounts for the sandbox node selector, cordons, the placement block, the node-group candidate set, an evacuation's cohort pin, and taints the pod doesn't tolerate. A fleet with no such node, or a cold cache, keeps today's unconstrained placement instead of leaving the restore Unschedulable.
• Capture CPU. It's the snapshot's recorded host_cpu. A snapshot with no recorded CPU falls back to the live label of the node that uploaded it, but only when that Node object is older than the snapshot, since a replacement instance can reuse the name. A label that isn't in canonical form reads as unlabeled, which matches how kube-scheduler compares it.
• Shortlist. The source-aware shortlist (SPACE_SANDBOX_RESTORE_SOURCE_NODE_HARD_PIN) drops fallbacks labeled with an excluded CPU. When the source's own CPU is excluded, the node-services readiness filter isn't allowed to leave it with no compatible fallback.
• Stranded-restore redirect. A failed cross-node restore leaves the sandbox pinned to the node that couldn't load it, so every later wake is a local-only restore on that node. Such a local-only plan switches to the snapshot's storage URI when all of these hold:
  • the pinned node's CPU can't load the snapshot, and a reachable compatible host exists;
  • the plan didn't come from the active execution's pod;
  • no pod of the sandbox could be hosting a live VM;
  • the sandbox has no local artifact marker;
  • the snapshot covers the last recorded command and access;
  • no newer full snapshot is still uploading or has failed;
  • for a restoring row, no full snapshot postdates the restore's start.
• Compatibility list. SPACE_SANDBOX_RESTORE_HOST_CPU_COMPATIBILITY takes comma-separated one-way capture=>loader pairs, letting a known-compatible pair share restores. An invalid value fails startup.

Gate-independent:

• node-ctl. node-ctl now publishes the host-CPU label before it advertises a node's network-slot capacity. Sandbox pods request a network slot, so a new node doesn't take one until it's labeled, unless the label write fails.

Code:

• crates/libs/api/src/controllers/substrate/kubernetes/pod_ops/host_cpu.rs (requirement, exclusions, redirect);
• store/snapshot_capture.rs (capture-CPU evidence queries);
• the call sites in pod_ops/{create_request,ensure_pod,redrive,restore}.rs;
• the shortlist filter in mod.rs;
• the pod builder in crates/libs/common/core/src/k8s.rs.

Known limits:

• The excluded CPUs are fixed when the pod is built. A node whose CPU model the fleet has never published, joining while a restore waits, isn't excluded, and the restore can bind there and fail.
• A node whose label write fails still advertises capacity and reads as unlabeled, so a restore can land there.
• The label fallback for a snapshot with no recorded CPU uses the uploading node. That's the capturing host for a suspend snapshot, but not for a copy one node pulled and another backed up.
• A legacy snapshot whose uploading node is gone stays unconstrained.
• The redirect applies on the next wake after the failed restore's execution is retired. While that restore's pod is still alive, the plan comes from the pod and isn't redirected.
• The redirect only acts on rows without a local artifact marker. The drain and gone-node give-ups clear the marker of a row a stuck restore left pinned; a stranded row that keeps its marker keeps today's local-only plan.

Test plan

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --locked -- -D warnings
• cargo nextest run --workspace on a Linux build host, including the Postgres-backed tests
• New tests:
  • restore_requires_the_capture_cpu_only_when_a_schedulable_host_has_it, including the gate-off case
  • a_restore_stranded_on_an_incompatible_cpu_restores_from_its_snapshot, including the gate-off case
  • restore_requirement_counts_only_hosts_inside_the_node_group
  • restore_requirement_counts_only_hosts_whose_taints_the_pod_tolerates
  • evacuation_requirement_counts_only_hosts_in_the_target_cohort
  • restore_requirement_falls_back_to_the_capturing_nodes_label
  • a_listed_compatible_cpu_takes_the_restore
  • restore_shortlist_drops_fallbacks_on_another_cpu
  • a_non_canonical_host_cpu_label_reads_as_unlabeled
  • taint_toleration_matching_follows_kubernetes_rules
  • pod snapshots sandbox_pod_restore_host_cpu_shortlist and sandbox_pod_restore_host_cpu_storage_only
• Mixed-CPU validation. A Kind cluster's nodes share one host CPU, so the cross-CPU path needs a cluster with workers of more than one CPU model. It happens per environment when the gate is turned on.

Docs

• docs/configuration.md: SPACE_SANDBOX_RESTORE_HOST_CPU_PLACEMENT_ENABLED (including the unpublished-CPU limit) and SPACE_SANDBOX_RESTORE_HOST_CPU_COMPATIBILITY.

[nikhilbirmiwal-pplx]

A few suggestions:

1. This is a non-trivial patch touching a critical feature, I would suggest we split it into a logical stack: RFC -> Protobuf Changes -> NodeChanges. ->K8s Changes.
2. I don't see mention of testing on a cluster with mixed nodes, is that something we can do?
3. Do we have any feature flags or env-gates to ensure smooth rollout?

[nikhilbirmiwal-pplx]

Is there value in checking in these svgs / pngs / mmds into git history? I am thinking they will accumulate and bit rot over time.

[the-gigi-pplx]

They reflect the history. If they are not useful I can just drop them.

[nikhilbirmiwal-pplx]

Does this make Linux based assumptions? I think in Rust there are flags which allow us to compile features based on target system, did we do that there?