[ac-pplx]

Summary

Mechanical, behavior-preserving split of the three largest files of the api Kubernetes substrate (crates/libs/api/src/controllers/substrate/kubernetes/) into concern-scoped submodules, extending the existing store/, adapter/ and pod_ops/ sibling directories. There is one commit per source file. Function bodies, signatures, SQL text after the sql! locator comment, error/log/metric strings and tests are unchanged; the changed locators and tracing metadata are listed under "Observable changes". No file outside the three modules changes, and the test files themselves are untouched.

File	Before	Module root after
store.rs	5,561	674
adapter.rs	4,890	77
pod_ops.rs	4,774	378

What moved where

Each new file starts with a //! doc naming what it owns and pulls in its parent's namespace with use super::*;. Names therefore resolve to the same items, traits and macros as in the original module. Items move whole, contiguous and in their original order. Moved inherent methods sit in an impl block per original block.

store/ (commit 1)

The module root keeps the store vocabulary: K8sStore, every row/outcome/claim type with its Display/FromStr/FromRow impls, the shared free helpers (classify_activation_route_publication, normalize_pod_uid, record_route_identity_decision, recovery_lease_ttl_secs, emit_started_suspend_recovery_event) and the constructor/accessor methods. These types stay put because their fields are private or pub(super). Moving them would widen every field, and keeping them means the moved methods use them unchanged.

New module	Contents (impl K8sStore methods unless noted)	Lines
store/execution_rows.rs	sandbox_k8s_executions row claims, locks, finishes and reads (execution_pod_name … ensure_active_execution_claim_in, replace_active_execution_claim_if_generation, ensure_active_execution_row, finish_active_execution_if_generation, the abandon_*unmaterialized_create_claim* family, locked_lease_owner_matches, reset_active_execution_to_creating_if_generation, active_execution_is_generation, active_execution_status)	1,099
store/started_recovery.rs	started-recovery lease, restore context, replacement, completion and observation	598
store/restore_lease.rs	restore ownership snapshot/observation, rollback revert, begin_restore_lifecycle, markerless restore lease, release_user_restore_lease, prepare_wake_sandbox	578
store/activation_fence.rs	activation attempt assign/complete/release/abandon, legacy route publication, clear_command_ready_pending, reconcile_activation_fence	580
store/pod_placement.rs	pod metadata, selected route pod IP, pod materialization claim, node placement persistence	282
store/node_inventory.rs	node inventory persistence, demotion, upsert_k8s_node_role, free fn is_node_k8s_tombstoned	301
store/restore_source.rs	restore source node and suspend/stopped storage-URI selection (artifact pointer reads)	267
store/suspend.rs	suspend CAS: finish/finalize suspended execution, persist_or_accept_suspend_commit, claim_suspend_execution, backup pending, committed artifact, missing suspending-pod finalization	634
store/pod_tracking.rs	forget_*pod_tracking*, backfill_missing_pod_uid, failed-recovery pod-absence convergence, clear_pod_tracking_in and the started-suspend recovery helpers it drives	612

adapter/ (commit 2)

The module root keeps the imports and constants. It re-exports what siblings, the kubernetes module and the adapter tests name.

New module	Contents	Lines
adapter/pooled.rs	warm-pool adoption: route-identity/wake requests, split-receipt capability wait, reclaimable metadata-update classifier, admitted_pool_pod_placement_admits, apply_pooled_execution_auto_lifecycle_with_client, apply_adopted_per_sandbox_setup	431
adapter/suspend.rs	suspend flow: await_suspend_continuation, suspend_execution_tracked, await_generation_suspend, poll_generation_suspend, RPC retry classifier, published artifact, pod-deletion confirmation, finish_suspend_handoff	417
adapter/node_eligibility.rs	NetworkSlotCapacityPreflight, node readiness/schedulability/placement-block/network-slot predicates and preflight, preservable_create_node, try_resolve_node_api_endpoint	256
adapter/teardown.rs	delete flow: fence types (FenceReadFreshness, FencedTeardown, PodFenceCheck), fenced reads, teardown_execution, lost-tracking classification, no-active-execution retire	723
adapter/substrate_adapter.rs	impl SubstrateAdapter for KubernetesSubstrate, moved as one block	3,044

adapter/substrate_adapter.rs is over the ~2,000-line goal. A trait impl can't span modules, so splitting it further means delegating each trait method to an inherent method elsewhere. That changes function bodies, which this pure move forbids (see open questions).

pod_ops/ (commit 3)

The module root keeps the shared materialization vocabulary: PodCreateContext/RestoreContext/SandboxPodRequestInputs with parse_pod_create_context, StartedRecoveryOwnership, ActivationAttemptAssignment, PodMaterializationOwnership, EvacuationContext, the kube error mappers, and two structs with private fields used by more than one path (RoutablePodIdentity, StartedRecoveryPodReplacement).

New module	Contents	Lines
pod_ops/delete.rs	throttle-retrying deletes and their constants, lingering_sandbox_pod_names, UID-preconditioned blocking deletes, delete_pod_and_forget_tracking	347
pod_ops/read.rs	lookup_sandbox_pod, cached/uncached pod reads, remember_pod_node_placement, resolve_reachable_pod_ip, resolve_pod_ip and its terminal/succeeded-pod helpers	455
pod_ops/redrive.rs	RedriveOutcome, redrive metric/claim guards (with their Drop impls), PreconditionReread, await_scheduling_outcome, redrive_unschedulable_restore_cross_node(_recorded)	773
pod_ops/restore.rs	SuspendRestorePlan, ExecutionHydrationOutcome (still pub, re-exported), routability checks, hydrate_restored_execution, suspend/stopped restore plans	336
pod_ops/started_recovery.rs	started-recovery replacement event details, source-node check, handoff, replace_started_recovery_pod	173
pod_ops/worker_clients.rs	pod worker/internal clients and channel, proxy TCP probe, builder node client	177
pod_ops/wake.rs	SelectedWakeOutcome, WakeDispatchStatus, ActivationWakeFailure, SelectedWakeSettings, selected wake pod IP/binding, execute_activation_wake_action, wake_selected_execution	592
pod_ops/create_request.rs	effective network policy, node-group placement resolution, sandbox_pod_request, pod_create_context_for_claim	309
pod_ops/ensure_pod.rs	ensure_pod_exists, restore_placement_sample, require_observed_create_claim_in_flight, stale-pod reason and cleanup	1,328

Move check

python3 move_check.py origin/main (the multiset script from the task) over the whole PR:

moved lines: 13954   removed-only: 202   added-only: 227

Every residual line falls into one of these classes:

Class	Removed	Added	Why
Visibility on moved items: pub(super) → pub(in crate::controllers::substrate::kubernetes)	114	114	pub(super) names the parent one level deeper after the move. The explicit path keeps the original scope.
Visibility on moved items: private → pub(super)	60	60	The item is used by a sibling submodule or the module's tests. pub(super) from the child equals the original private scope (the original module and its descendants).
super:: paths in moved code, one level deeper (super::X → super::super::X, super::super::X → super::super::super::X, one intra-doc link)	28	28	Same targets (kubernetes constants and helpers, PublishedLocalArtifact, ActivationLeaseCompletion, …).
rustfmt reflow caused by the longer visibility/path text (&self,, ) -> bool {, node: &Node,, F,/D,/>(, == target_state.as_str(), …)	0	20	Signature lines that crossed 100 columns. The removed side counts under the visibility rows.
Continuation lines of multi-line use re-export lists in the module roots	0	5	The checker treats only a use statement's first line as structural.

There are no other residual lines. In git diff --color-moved=zebra --color-moved-ws=allow-indentation-change origin/main...HEAD, everything outside these classes shows as moved except a few short fragments (self.error, a closing brace) where a rewritten visibility line splits a block below git's 20-character moved-block minimum. The multiset check matches those fragments.

Visibility changes

No item became pub that wasn't already, and no struct field changed visibility. Every change below preserves the item's original scope after it moved one module deeper.

• pub(super) → pub(in crate::controllers::substrate::kubernetes) (114 lines): every pub(super) item, method, enum or field that moved into a submodule. That covers all moved pub(super) K8sStore methods; adapter pooled_route_identity_adoption_request, generation_suspend_rpc_should_retry, published_generation_suspend_artifact, suspend_pod_deletion_confirmation, finish_suspend_handoff, NetworkSlotCapacityPreflight, node_schedulable, node_not_placement_blocked, node_services_not_ready, node_network_slot_available and network_slot_capacity_preflight; and pod_ops RedriveOutcome, SuspendRestorePlan, SelectedWakeOutcome/WakeDispatchStatus/ActivationWakeFailure (with their pub(super) fields and methods) plus the moved pub(super) KubernetesSubstrate methods.

• private → pub(super) (60 items):

  • store (23): execution_rows::{execution_pod_name, active_execution_row, insert_active_execution_claim_in, update_active_execution_pod_metadata, lock_active_execution_state_in, update_active_execution_node_name, finish_active_execution_in, identity_binds, finish_fenced_execution_in, finish_active_execution_generation_in, finish_active_execution_generation_attempt_in, lock_sandbox_for_execution_mutation_in, lock_sandbox_execution_lifecycle_in, lock_sandbox_for_execution_lifecycle_in, ensure_active_execution_claim_in}, restore_lease::restore_ownership_collision, activation_fence::assign_activation_attempt_in, node_inventory::upsert_k8s_node_role, suspend::suspend_owns_pod_teardown_in, pod_tracking::{clear_pod_tracking_in, recover_markerless_restoring_after_execution_cleanup_in, recover_started_after_suspend_cleanup_in, recover_started_after_failed_recovery_in}.
  • adapter (15): pooled::{pooled_wake_request, wait_for_pooled_split_receipt_capability, pooled_metadata_update_error_is_reclaimable, admitted_pool_pod_placement_admits, apply_pooled_execution_auto_lifecycle_with_client, apply_adopted_per_sandbox_setup}, suspend::{await_suspend_continuation, await_generation_suspend, poll_generation_suspend}, node_eligibility::preservable_create_node, teardown::{FenceReadFreshness, FencedTeardown, PodFenceCheck, check_pod_fence, teardown_execution}.
  • pod_ops (22): delete::{K8S_THROTTLE_DELETE_RETRY_ATTEMPTS, retry_kube_delete_throttling, delete_sandbox_pod_with_throttle_retry, lingering_sandbox_pod_names, delete_lingering_prior_generation_pod, delete_sandbox_pod_for_redrive, k8s_blocking_delete_sandbox_pod_for_operation}, read::{terminal_cleanup_phase_after_uncached_check, succeeded_pod_publication_grace_remaining}, redrive::{PreconditionReread, classify_precondition_reread, await_scheduling_outcome, redrive_unschedulable_restore_cross_node_recorded}, restore::{pod_matches_execution_generation, restored_pod_routability, stopped_restore_plan}, started_recovery::{started_recovery_replacement_event_details, replace_started_recovery_pod}, create_request::{effective_network_policy_for_sandbox, resolve_node_group_placement_for_claim, pod_create_context_for_claim}, ensure_pod::restore_placement_sample.

  Each widening was proven necessary: reverting it alone and running cargo check -p pplx-space-api --all-targets --all-features fails. Two were found by the compiler rather than by name search: admitted_pool_pod_placement_admits (inside a wholly moved impl block) and FencedTeardown (named only through a type it is a field of).

• Re-exports in the module roots keep every existing path working. They use the original visibility (pub(super) use, pub use for ExecutionHydrationOutcome, private use for module-internal names). Names used only by the module's tests are re-exported under #[cfg(test)] so non-test builds have no unused imports.

Observable changes

Code bodies are unchanged, but two compile-time call-site markers derive from module_path!()/file!()/line!(), and they now reflect each moved item's new module and line. No explicit target: arguments or locator overrides were added.

sql! locators

pplx_space_common_core::sql! prepends /*module_path!():line!()*/ to each statement.

• 90 moved sites: 89 from store.rs, 1 from adapter.rs (none in pod_ops.rs). Their locator now names the child module and new line, e.g. /*pplx_space_api::controllers::substrate::kubernetes::store::suspend:NNN*/ instead of …::kubernetes::store:NNN.
• 1 site stays in store.rs (template_network_policy). Its module path is unchanged, but its line shifts from 646 to 656 because of the inserted mod declarations.
• Where locators are visible: pg_stat_activity, slow-query logs, Datadog DBM samples, and the db.statement span attribute.
• Span names are unaffected: the sqlx OTel span name strips the locator (crates/libs/observability/src/sqlx_otel.rs).
• The SQL after the locator is byte-identical.
• Prepared-statement cache: sqlx's per-connection cache sees new statement keys. Caches start empty on deploy, so no lasting effect is expected.
• pg_stat_statements: queryid ignores comments, so per-statement aggregation is unaffected.

tracing call sites

The 117 moved call sites are 109 events and 8 tracing::info_span! spans. The spans are in adapter pooled (3) and substrate_adapter (4), and in pod_ops/redrive.rs (1). The three original files had 12 (store), 60 (adapter) and 45 (pod_ops) call sites, and all moved; the module roots keep none. Events per new module:

• store: started_recovery 2, activation_fence 3, node_inventory 1, restore_source 1, suspend 1, pod_tracking 4.
• adapter: pooled 4, suspend 6, teardown 8, substrate_adapter 35.
• pod_ops: delete 2, read 9, redrive 13, wake 1, create_request 4, ensure_pod 15.

None of the moved code uses #[instrument] or an explicit target:. SandboxEvent::emit_with records are emitted from the events crate, so their target doesn't change.

For both events and spans:

1. Targets change. The default target is module_path!(), so records now carry the child-module target, e.g. pplx_space_api::controllers::substrate::kubernetes::pod_ops becomes …::kubernetes::pod_ops::ensure_pod, and …::kubernetes::adapter becomes …::kubernetes::adapter::substrate_adapter.
2. File, line and module metadata change. So do the exported code.namespace, code.filepath and code.lineno attributes, and default event names, which embed file:line.
3. Explicit span names, messages, fields and levels are unchanged.
4. Existing filters still apply. RUST_LOG/EnvFilter directives match targets by prefix, so a directive for pplx_space_api, …::substrate::kubernetes, or any of the three original module paths still selects these records.
5. Exact-target references: none found. I searched infra/, docs/ and scripts/ for @target, target= and the module paths (controllers::substrate, substrate::kubernetes, pplx_space_api::controllers, kubernetes::{store,adapter,pod_ops}), then all non-Rust files and Rust string literals repo-wide for the module paths. The target= hits are unrelated (Tempo/Loki -target=all, Terraform -target=, shell/Python variables, netgtw probe targets). No test filters on or asserts a tracing target.

Hazard self-check

• Globs: each new file has exactly one glob, use super::*, and its only local items are the ones it moved. Those were local to the original module too, so no name now resolves to a different item. The roots' import lists are unchanged.
• Trait method resolution: children see exactly the root's traits. K8sStore implements only Clone, so a private inherent method can't fall back to a same-named trait method. The adapter/pod_ops inherent method names don't collide with SubstrateAdapter methods.
• cfg gates: the three #[cfg(test)] K8sStore methods moved with their attributes. #[cfg(test)] mod tests; stays where it was in each root. Checked with --all-features --all-targets and the default-feature build (where the kubernetes module is gated off).
• Paths and macros: no include_str!/include_bytes!/env!/sqlx::query! paths or macro_rules! in the moved code. sql! and tracing call-site metadata (module_path!/line!) do change; see "Observable changes".
• Structs: no struct moved with a changed field list and no fields were reordered, so Drop order is unchanged.
• Snapshots: the store tests module path (store::k8s_store_postgres_tests) is unchanged, so insta snapshot names are unchanged.
• sqlx offline data: the crate uses no compile-time-checked sqlx::query! macros and the repo has no .sqlx offline data, so there's nothing to regenerate.
• Rustdoc: cargo doc -p pplx-space-api --no-deps --document-private-items --all-features gives the same 21 warnings as the base commit, so no intra-doc link broke.

Validation

Command	Result
cargo fmt --all --check	pass
cargo clippy -p pplx-space-api --all-targets --all-features --locked -- -D warnings (after every commit)	pass
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings (CI clippy tier)	pass
cargo clippy -p pplx-space-api --all-targets --locked -- -D warnings (default features, kubernetes module gated off)	fails on 2 errors in untouched files (substrate/mod.rs unused pod_names, fork_operation_executor.rs unconstructed SubstrateNodeId). Identical on base 15a2209, so pre-existing; CI does not lint this configuration.
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run -p pplx-space-api --all-features --all-targets --run-ignored all --no-fail-fast (nextest Postgres + Redis setup scripts)	1657 passed, 0 skipped; Postgres/Redis setup PASS; dependency-skip log empty; all 23 store::k8s_store_postgres_tests ran and passed
cargo test -p pplx-space-api --all-features --doc	ok (the crate has no doctests)
cargo doc -p pplx-space-api --no-deps --document-private-items --all-features vs base	same 21 warnings as base; no new broken intra-doc link
python3 move_check.py origin/main	residual fully classified above
rustfmt --edition 2024 --check scripts/generate-env-contract.rs, scripts/generate-env-contract.sh --check, scripts/check-rust-error-policy.sh (the CI fmt step's guards)	pass
pre-commit run --all-files (gitleaks)	pass
scripts/check_no_inline_rust_test_modules.py, scripts/source_guards/thin_service_mains.py	fail on untouched files (crates/libs/observability, crates/apps/*/main.rs); output byte-identical on base, so pre-existing. The other scripts/source_guards/*.py pass.
sqlx offline data	N/A: no sqlx::query! macros and no .sqlx directory in the repo

Kind

Environment: the repo Kind stack (infra/space/kind/scripts/bootstrap.sh, then Tilt through scripts/tests/smoke-kind-e2e.sh) on a Linux/KVM devbox, with CI's smoke settings SPACE_KIND_OBSERVABILITY=off SPACE_KIND_TELEMETRY_MODE=datadog.

Deployed binary is this branch. HEAD ca4032b with a clean tree. The Kind build's source mirror (~/.space-devbox/kind/rust-src/crates/libs/api/src) is byte-identical to the worktree. The space-api Deployment ran image space-api:tilt-e1e07d92c1afca65 (ID sha256:e1e07d92c1af…, built after the last commit). sha256(/space/bin/space-api-svc) in the running pod was 9a176c0b6dfd…, equal to the staged artifact. The binary embeds source paths that exist only on this branch (substrate/kubernetes/pod_ops/ensure_pod.rs, adapter/substrate_adapter.rs, store/pod_tracking.rs, …).

Quiet gate. Tilt reported no pending or in-progress builds and no ReplicaSet younger than 180 s, followed by a 120 s settle, before the suite started.

Run	Result
smoke-kind-e2e.sh (bring-up baseline)	PASS
smoke-kind-typed.sh all, ECW pass	72 passed, 0 failed, 3 inconclusive/infra errors
smoke-kind-typed.sh all, native pass	23 passed, 0 failed, 0 inconclusive
Rerun ecw.cold_create_environment, ecw.warm_pool_environment	PASS, PASS
Rerun local.encrypted_resuspend_preserves_cow	INFRA_ERROR again (same signature)

Classification of the first-pass non-passes:

• ecw.cold_create_environment, ecw.warm_pool_environment: ENVIRONMENTAL. Both were inconclusive because the run lacked the receipts-directory input (SPACE_SMOKE_KIND_COLD_RECEIPTS) that kind-ci.yml provides. Rerun with the CI-equivalent input, both pass.
• local.encrypted_resuspend_preserves_cow: PRE-EXISTING. The scenario's assertions pass, but its critical cleanup fails: the deleted sandbox's pod is still Running 60 s after delete. The pod carries a deletionTimestamp (the api issued the delete), and the node reports FailedKillPod … KillPodSandbox … DeadlineExceeded, i.e. the node runtime fails to tear the VM down. The rerun on this branch failed the same way. I then deployed base 15a2209 to the same cluster from a separate worktree (binary hash 40ca0c0b07ea…, none of the new module paths) and the scenario failed with the identical signature.

No CHANGE-INDUCED failures.

Open questions

1. adapter/substrate_adapter.rs (3,044 lines) holds the whole SubstrateAdapter impl. Getting under ~2,000 lines needs thin delegating trait methods whose bodies live in the concern modules, e.g. create_execution (≈650 lines), claim_pooled_execution (≈560) and prepare_pooled_execution (≈300). That changes function bodies, so it's left for a follow-up that allows it.
2. pod_ops/ensure_pod.rs (1,328 lines) is dominated by the single 1,181-line ensure_pod_exists. Splitting it needs body changes.
3. local.encrypted_resuspend_preserves_cow fails its cleanup on main too: a node-side KillPodSandbox deadline leaves the pod Terminating. It's unrelated to this PR but worth a separate look.
4. Moved pub(super) items use the explicit pub(in crate::controllers::substrate::kubernetes), as the task asks. pub(in super::super) would be shorter and cause fewer rustfmt reflows, if reviewers prefer it.

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 2 findings

Low

1. Four comments outside the three modules now name a file that no longer holds the item. The PR leaves every file outside the modules untouched, but these comments point at the old flat files:

   • crates/libs/cluster/controller/server/src/node_drain/mod.rs:3240: upsert_k8s_node_role "in …/kubernetes/store.rs" now lives in store/node_inventory.rs:100.
   • crates/libs/sandbox/portability/src/lib.rs:1568: the ORDER BY sn.timestamp DESC LIMIT 1 selector "in kubernetes/store.rs" is now in store/restore_source.rs:112.
   • infra/space/kind/charts/space-node-stack/templates/node-fence.yaml:139: node_network_slot_capacity "in substrate/kubernetes/adapter.rs" is now in adapter/node_eligibility.rs:75.
   • space_smoke/template_build.py:16: builder_client_for_node "in substrate/kubernetes/adapter.rs" is now in adapter/substrate_adapter.rs:1806.

   None of them breaks anything, but each one now sends a reader to a file that no longer holds the code. Naming the module (kubernetes::store, kubernetes::adapter) rather than a file would survive the next split. The chart comment can wait for a later chart change if you'd rather not touch a template in a pure-move PR.

2. PR description: the move check cites move_check.py "from the task", which isn't in the repo, so a reader can rerun only the --color-moved view, not the multiset check behind the residual table.

I'm not counting adapter/substrate_adapter.rs (3,044 lines) as a finding. You've already explained why a trait impl can't move further without changing function bodies, and that belongs in a follow-up.

Sequencing, not a defect: eight other open PRs touch kubernetes/{store,adapter,pod_ops}.rs (#221, #176, #168, #152, #151, #150, #128, #3). Whichever of those lands after this one has to re-port its hunks into the new files. PR #259 doesn't overlap.

Verified against ca4032b248ea. My own checks, all run against the fetched ref with no cargo:

• Multiset of removed vs. added lines: 202 removed-only and 389 added-only. With pub(in crate::controllers::substrate::kubernetes) and pub(super) normalized and super:: chains collapsed, 9 and 196 remain. Every one of those is a mod or use re-export line (including its continuation lines), a //! header, an impl K8sStore { / impl KubernetesSubstrate { opener with its closing brace, one of the 8 #[cfg(test)] re-export gates, or a rustfmt reflow of a signature that crossed 100 columns.
• Order: a token-level ordered alignment of every new file against its source file leaves 0 unmatched tokens in 18 of 23 files. The other 5 differ only by the trailing commas rustfmt adds to reflowed signatures.
• Attributes: no impl block in the three base files carried an attribute, so splitting them drops none.
• Tests: CI ran 8,681 tests, 1,657 of them in pplx-space-api. The base commit's run on main ran exactly the same numbers, all passing.

I relied on CI for compile, clippy and the test run. I didn't rerun your Kind suites or the per-widening cargo check reverts.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.