[the-gigi-pplx]

Summary

Adds the first production writer of the cluster's execution environment policy: admin API routes under /admin/api/v1/runtime-environments, and matching space-admin-cli runtime env commands. Until now the policy store had no caller that writes it.

Route	CLI	What it does
GET /runtime-environments	runtime env list	The policy: generation, default, legacy permission, and each environment's state, with who last changed each row and why.
PUT /runtime-environments/default	runtime env default <ref> --reason [--strand]	Makes an environment the default. It becomes allowed if it isn't; the old default stays allowed.
PUT /runtime-environments/{ref}/state	runtime env state <ref> <state> --reason [--strand]	Sets allowed, deprecated or blocked.
GET /runtime-environments/nodes	runtime env nodes [--after ID | --all]	Each node's published execution policy compared with the cluster policy, 100 nodes per page.
GET /runtime-environments/population	runtime env population	Aggregate counts of sandboxes, by lifecycle, and template releases per environment binding.

Both writes go through pplx-space-runtime-environment-policy-store, which locks the policy row, enforces its rules and increments the generation in the same transaction. A write it refuses is a 409 that names the rule: blocking or deprecating the default, blocking with no default set, or making a blocked environment the default. A required reason (1 to 1024 characters) and the caller are recorded on the rows the write touches, and each write emits a space.admin.audit event as well as the existing request audit row.

The API adds one check of its own, for a write that stops an environment executing. Blocking stops the blocked environment, and the cluster's first default stops every environment with no state, since nodes then narrow to the environments that have one. Such a write needs strand: true while sandboxes, or template releases still in use, are bound to an environment it stops, or while any sandbox's environment can't be told: no binding, an unreadable one, or no binding record. Unbound legacy sandboxes count because they execute as the environment they match, which only a node's compiled catalog can resolve, and a sandbox with no observation from its node may be bound to one that stops. A release is in use while a tag points to it or an enabled pool config pins it. An in-use release counts when it's bound to an environment that stops or its environment can't be told; a superseded release is reported but doesn't count, since it strands work only through the sandboxes made from it, which do count. Sandboxes that can't run again (cleanup_required or deleting, or failed with no live evidence and nothing the repair loop could revert to, by the shared failed_repair_revert_eligibility rule) are reported but don't count either. Later defaults only add themselves as allowed, and blocking an environment that already doesn't execute (blocked, or with no state under a default) stops nothing new, so neither is checked. A concurrent write between that read and the store's lock is caught by the generation the write returns, and the check then runs under the lock. The counts are read just before the write's transaction, so they're an acknowledgement, not a guarantee.

Node status reports the generation each node applied and whether it's current, and judges conformance by content, so a node behind on generation still conforms when nothing it would change has changed. A node conforms when its default matches, it executes nothing the cluster doesn't, and it doesn't allow legacy work the cluster revoked. A node that narrows further than the cluster, or refuses legacy work the cluster permits, still conforms.

Population attributes a suspended or restoring sandbox to the snapshot its substrate's restore would pick. On Direct that's the newest full snapshot in any upload state, since restore runs the source node's artifact. On Kubernetes it's the newest ready full snapshot with a live storage location: while a node drain evacuates the sandbox, the one holding its current local artifact or, with no artifact marker, the newest full snapshot; otherwise, for a suspended sandbox, the one holding its current local artifact or, with no marker, one taken since the last placement, and for a restoring sandbox, the newest from before the restore began. The query interpolates the shared snapshot_is_current_local_artifact, restore_retry_source_snapshot_is_current and no_newer_full_snapshot predicates rather than re-spelling them. A failed sandbox with evidence the repair loop reads as live (a non-failed status, a pid, a node assignment, or an active Kubernetes execution) is attributed by its observation, or counted as unrecorded without one; otherwise a failed Kubernetes sandbox the repair loop can revert is attributed to the snapshot it would then restore from, and any other failed sandbox has no record. Every other sandbox is attributed to the execution metadata its node last published. Deletion tombstones aren't counted. Work with no binding record and work with no environment in its record are reported in their own buckets.

Supporting changes:

• The policy store gains get_record, which reads the policy with each row's last change. get reads through it, so there's one read path.
• The store exports RuntimeEnvironmentState::ALL and PolicyChangeReason::MAX_CHARS, which the API errors and the OpenAPI document use.
• AdminNodeRow::execution_policy decodes a node's stored policy once, for the node detail route and node status alike.
• support::read_db_error answers a read's database failure without the write-race 409.
• AdminClient::send is the CLI's shared send-and-interpret helper.

Validation

• cargo fmt --all -- --check, and cargo clippy --all-targets --all-features -D warnings on the three changed crates.
• cargo arch: no findings in the changed files.
• Route tests drive the real admin router, with session auth and the audit middleware, against a migrated Postgres. They cover each verb, every store refusal, malformed bodies and paths, the strand gate on blocks and on the first default, including the store's refusal taking precedence, releases that are superseded, tagged, or pinned by an enabled or disabled pool config, failed sandboxes that are live, repairable, unrepairable, or on Direct, authorization, node paging at exactly 100 and 101 nodes, re-blocking an already-blocked environment, including one another operator allows again while the block waits for the policy lock, and a population fixture covering each binding source on both substrates, the artifact marker, restore retries, orphaned and missing storage locations, snapshots from a previous placement, and tombstones.
• CLI tests run against a local stub server: request shape, rendering, error propagation, --all paging, and argument parsing.
• cargo mutants on the new store, API and CLI code: every viable mutant is caught except the one that drops the space.admin.audit event, which only a log assertion could catch, and the test policy rules those out. cargo-mutants doesn't mutate SQL, so each arm of the population query's snapshot filter was removed by hand: the population test fails for every one.
• ci/scripts/run-rust-checks.sh tests-unprivileged on a Linux devbox at this head: 10,535 passed, 55 privileged tests skipped, no dependency skips. The privileged tier passed except three fs-verity tests that need a kernel with fs-verity, which the devbox lacks; this change doesn't touch them.

Severity and risk

Severity: does not apply; this is new operator surface.

Risk: nodes apply the cluster policy the cluster API serves them, and it serves the policy only while SPACE_INTERNAL_API_RUNTIME_DESIRED_STATE_ENABLED is on, which is off by default. Where it's on, a mistaken block stops new work on that environment, which is what the strand check guards. The fix is another PUT .../state, which is audited. The only existing behavior touched is the node detail route's policy decode, which moves into AdminNodeRow::execution_policy unchanged.

Not covered

• Setting a default doesn't check that every node can run it. That needs per-node usable environments, which have a column but no writer yet; a node that applies the policy is to refuse a default it hasn't installed and verified.
• Blocking doesn't check for a finished template campaign, which doesn't exist yet, or refuse while nodes can't enforce a block; node status shows each node's applied generation, and refusing on it is a later change.
• A policy write doesn't trigger node reconciliation. The runtime-provisioning facet reads desired state on its periodic tick.
• No route sets the legacy unbound permission, though the store supports it.

AI assistance: Claude Code (Claude Opus) drafted the code, the tests and this description.

[johnlouie-pplx]

Approving. I read the handlers, the population query, and the store changes, and checked that the CI Rust tests job ran the Postgres-backed write and auth tests rather than skipping them.

One non-blocking follow-up:

• Non-blocking: policy writes have no generation precondition. PUT /runtime-environments/{ref}/state and PUT /runtime-environments/default take the policy row lock but never compare against the generation the operator last read. So if one operator blocks an environment while another runs set-state <ref> allowed from an older list, the second write reopens the environment and overwrites the block reason. Re-allowing skips the strand check, so nothing stops it. update_template already handles this for network policy: an optional If-Match: <generation> header, checked in the write, with a 412 on mismatch. Could both PUTs take the same header, compared after lock_policy, by moving parse_if_match_generation into support.rs? The CLI could then send the generation it just read, or take --if-generation.