Skip to content

node: lease exclusive devices through the fenced sandbox lifecycle - #947

Draft
zibi-pplx wants to merge 1 commit into
pr945from
pr946
Draft

zibi-pplx wants to merge 1 commit into
pr945from
pr946

Conversation

@zibi-pplx

@zibi-pplx zibi-pplx commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Keep typed launch validation and device-refusal coverage together; preserve GPU boot formats when consuming parsed adapter inputs.

Carry device requirements from standalone sandbox creation through durable sandbox rows, stopped-disk metadata, node allocation and the supervisor journal. Negotiate support and require assignment acknowledgements at each hop. Persist attaching intent before spawn, retain ownership after ambiguous outcomes, and prevent a delayed launch from bypassing a stop barrier.

Admit only catalog-compatible profiles and revalidate enrolled PCI/IOMMU identity. Reserve full guest RAM plus device overhead, configure memlock before privilege drop, and project only assigned VFIO nodes and PCI directories into the contained Cloud Hypervisor process. Build templates without devices while preserving kernel/initramfs bindings.

Enrollment begins quarantined. The optional gpu-development feature permits an explicitly authorized, host-boot-fenced single-T4 reset-only experiment. Release queues an independent pool worker; stop does not wait for reset. Checked reset evidence can return this development capacity to ready; uncertainty quarantines it. Production builds expose no development profile, and cluster-managed callers refuse device requests.

Validation

  • Rebase retains effective cluster-policy checks after admission waits and reads one effective policy for device preferences. Node-manager Linux ARM64 all-target Clippy passed after adapting the new policy API.

After the single-identity adapter cleanup, the launch-contract regression passed with device-bearing Firecracker requests still refused.

Generated sandbox Go bindings with CI's protoc 35.0; Go formatting, vet and tests passed.

The development lifecycle completed managed create, checked CUDA computation, stop/start with preserved disk data, repeated CUDA and delete. Focused ownership, memory, admission, downgrade, journal and CLI tests ran during implementation. Nine device-store tests and the capacity display test passed; Linux clippy passed for node-manager, supervisor and sandbox-manager consumers. Formatting and architecture checks passed (architecture advisories remain). Full CI remains pending.

Severity and risk

New privileged hardware ownership path, disabled without explicit development build and host authorization. Requires the preceding SQLite migration and device/boot contracts. Deploy supervisor and node-manager before enabling device callers; old peers must not silently accept CPU-only work. Database rollback is intentionally refused once device state exists.

Reset-only evidence accepts development risk; it does not prove erasure or qualify tenant reuse. GPU-held manager restart, unexpected VMM-death convergence, multi-device hardware and production fleet placement remain unvalidated. No running-node restart was performed for the capacity-reporting update.

Stack

GPU execution, after the CPU-only Cloud Hypervisor group. #962 is the first PR in this group and supplies the SQLite schema before #945 and the device consumers. Requirements, artifact delivery, fenced ownership, operator visibility and GPU smoke coverage follow. Device-bearing requests remain standalone-only. The scrub candidate in #950 is unqualified; this stack does not claim production-safe tenant reuse.

AI assistance: Codex implemented and validated the changes and drafted this description.


Stack created with Sapling. Best reviewed with ReviewStack.
Base: #960, the separate Cloud Hypervisor CPU execution stack.

This was referenced Oct 25, 2026
@zibi-pplx
zibi-pplx changed the base branch from main to pr945 October 25, 2026 16:55
@zibi-pplx
zibi-pplx added this pull request to stack #951 October 25, 2026 16:55
## Summary

Keep typed launch validation and device-refusal coverage together; preserve GPU boot formats when consuming parsed adapter inputs.

Carry device requirements from standalone sandbox creation through durable sandbox rows, stopped-disk metadata, node allocation and the supervisor journal. Negotiate support and require assignment acknowledgements at each hop. Persist attaching intent before spawn, retain ownership after ambiguous outcomes, and prevent a delayed launch from bypassing a stop barrier.

Admit only catalog-compatible profiles and revalidate enrolled PCI/IOMMU identity. Reserve full guest RAM plus device overhead, configure memlock before privilege drop, and project only assigned VFIO nodes and PCI directories into the contained Cloud Hypervisor process. Build templates without devices while preserving kernel/initramfs bindings.

Enrollment begins quarantined. The optional gpu-development feature permits an explicitly authorized, host-boot-fenced single-T4 reset-only experiment. Release queues an independent pool worker; stop does not wait for reset. Checked reset evidence can return this development capacity to ready; uncertainty quarantines it. Production builds expose no development profile, and cluster-managed callers refuse device requests.

## Validation

- Rebase retains effective cluster-policy checks after admission waits and reads one effective policy for device preferences. Node-manager Linux ARM64 all-target Clippy passed after adapting the new policy API.

After the single-identity adapter cleanup, the launch-contract regression passed with device-bearing Firecracker requests still refused.

Generated sandbox Go bindings with CI's protoc 35.0; Go formatting, vet and tests passed.

The development lifecycle completed managed create, checked CUDA computation, stop/start with preserved disk data, repeated CUDA and delete. Focused ownership, memory, admission, downgrade, journal and CLI tests ran during implementation. Nine device-store tests and the capacity display test passed; Linux clippy passed for node-manager, supervisor and sandbox-manager consumers. Formatting and architecture checks passed (architecture advisories remain). Full CI remains pending.

## Severity and risk

New privileged hardware ownership path, disabled without explicit development build and host authorization. Requires the preceding SQLite migration and device/boot contracts. Deploy supervisor and node-manager before enabling device callers; old peers must not silently accept CPU-only work. Database rollback is intentionally refused once device state exists.

Reset-only evidence accepts development risk; it does not prove erasure or qualify tenant reuse. GPU-held manager restart, unexpected VMM-death convergence, multi-device hardware and production fleet placement remain unvalidated. No running-node restart was performed for the capacity-reporting update.

## Stack

GPU execution, after the CPU-only Cloud Hypervisor group. #962 is the first PR in this group and supplies the SQLite schema before #945 and the device consumers. Requirements, artifact delivery, fenced ownership, operator visibility and GPU smoke coverage follow. Device-bearing requests remain standalone-only. The scrub candidate in #950 is unqualified; this stack does not claim production-safe tenant reuse.

AI assistance: Codex implemented and validated the changes and drafted this description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant