orchestrator: Move the cursor off a component gated mid-walk - #454
Conversation
59705ad to
8db5243
Compare
embediver
left a comment
There was a problem hiding this comment.
Tests look all good.
On the actual fix I'm not sure if I understood it completely but looks good otherwise.
rusty1968
left a comment
There was a problem hiding this comment.
Correctness (blocking)
The cursor-advance fix is applied only in PreSupervision's CorruptionDetected arm. The identical race is left open in AwaitingReady, which handles CorruptionDetected through handle_supervising's bare self.handle_corruption(*id, ctx) (lib.rs:773) — this call site was not touched by the diff.
AwaitingReady has the same chain[cursor]-keyed VerificationPassed release logic that motivated this fix in PreSupervision (lib.rs:598-621), and its cursor points at a real, currently-under-verification component by construction (that's exactly how a machine enters AwaitingReady — see lib.rs:517-528). Concretely: chain [C0: active_required, C1: passive_cascading, C2: passive_required]. BOOT; VerificationPassed(C0) → AwaitingReady(Some(C0)), cursor on C1 (VerifyFirmware(C1) already emitted). CorruptionDetected(C1) falls through AwaitingReady's match (no arm for it) to handle_supervising, which gates C1 via handle_corruption but never advances the cursor off it. A VerificationPassed(C1) already in flight — the same in-flight-verdict scenario this PR's own comment calls out — then matches chain[cursor] == C1 in AwaitingReady's own VerificationPassed arm and fires Effect::ReleaseReset(C1), taking a component the cascade just isolated and reported (ReportIsolated(C1)) back out of reset. That's the exact bug this PR fixes for PreSupervision, still live one state over.
Fix: factor the new logic in the PreSupervision arm into a shared method (e.g. handle_corruption_advancing) and call it from both the PreSupervision arm and handle_supervising's CorruptionDetected arm. This is a no-op for Ready/Updating (cursor is at the past-the-end sentinel there) and for Recovering (its own handler doesn't key release off chain[cursor]), so it's safe to apply uniformly. See suggested diff, which also adds a regression test mirroring the new PreSupervision tests but for AwaitingReady.
Verbosity/duplication (ponytail-review, non-blocking)
The "advance, then Handled if found else Transition(Ready)" pattern now appears a third time (PreSupervision's new corruption arm, PreSupervision's VerificationPassed, AwaitingReady's VerificationPassed). The suggested fix above naturally collapses one of those copies into the new shared helper; consider whether it's worth doing for all three, but that's optional and not required for this PR.
No issues found in the no-alloc / no-panic / secure-coding categories — the new code uses heapless/saturating_add consistently with the rest of the file, no unwrap/panics, no secret handling involved.
Test coverage for the three new PreSupervision scenarios is solid and follows existing conventions; it just needs a sibling test for the AwaitingReady path described above.
|
You were right. Asking why it was only The fix: Why its own arm rather than The other two are older than this branch. A Why a property rather than three more tests: four of these is a class. One more, in the cascade itself. The third copy I left alone: the other two are |
A cascade can gate the component currently under verification. Only chain[cursor] can be released, so leaving the cursor on that component let its in-flight verdict take it back out of reset. handle_corruption_advancing gates by policy and then moves the cursor to the next ungated component. The two states that release only chain[cursor], PreSupervision and AwaitingReady, route CorruptionDetected through it. It is not called from handle_supervising. Recovering's cursor is stale, since VerificationFailed leaves it on the failed component, so advancing there would verify mid-recovery or reach Ready instead of re-walking. The release property now randomizes the chain shape as well as the event sequence. The AwaitingReady race needs a gateable component after an active one, which the old fixed chain never had. Also adds the missing test for the Chain::try_from length bound. cursor is a u8 using chain.len() as its past-the-end sentinel, so a 256-entry chain would truncate that sentinel to 0 and the walk would never read as done. Assisted-by: Claude:claude-opus-5 Signed-off-by: Christina Quast <christina.quast@9elements.com>
A failure verdict or a corruption report can be in flight from before the cascade gated its component. VerificationFailed and CorruptionDetected now return early when the component is gated. Recovering it re-walks the chain for a device that stays held, and on exhaustion a Required one locks the platform down over a cascade that was already contained, which contradicts the rule that a non-Required cascade never reaches lockdown. property_isolation_is_sticky_under_random_sequences guards both: after ReportIsolated(id), nothing in the rest of the run emits ReleaseReset or RecoverComponent for that id. The cursor invariant the walk depends on is written down on the field, since that property is what guards it. Assisted-by: Claude:claude-opus-5 Signed-off-by: Christina Quast <christina.quast@9elements.com>
cascade_hold only queued a dependent it had just gated, so the walk stopped at a component isolated earlier and never reached what depends on it. Gate a middle node on its own, then corrupt its Cascading parent, and the component behind it comes out of reset with its whole dependency chain isolated. The frontier is the visited set now, so traversal continues through a gated component. gate_one is idempotent, so re-visiting one emits nothing. property_cascading_isolation_reaches_the_whole_subtree guards it. Assisted-by: Claude:claude-opus-5 Signed-off-by: Christina Quast <christina.quast@9elements.com>
92ab8df to
e19c5e5
Compare
Leon approved the PR. Rusty1968's changes were taken into consideration.
Two things, found by asking whether a cascade that fires mid-walk is covered.
The test:
mid_walk_cascade_skips_unreached_dependents. The existing cascadetests all corrupt a component after the walk reaches Ready, so the cursor is
already past everything the cascade gates, and the
is_gatedskip inadvance_to_next_ungatedis never exercised at depth. The new test corrupts C1while C0 is still under verification: C1 -> C2 -> C3 are gated before their turn
and none is ever read or verified.
The fix: writing that test showed that gating the component under verification
left the cursor on it, and release keys off
chain[cursor]alone, so aVerificationPassedalready in flight took an isolated component back out ofreset.
PreSupervision'sCorruptionDetectedarm now advances the cursor pasta gated component (Ready if the rest of the chain is gated). The late verdict is
then a cursor mismatch and is dropped, and the walk does not wait on a verdict it
must ignore.
Why advance in place rather than re-enter
PreSupervisionthe way therecovery-exhaustion path does: corruption already gates and continues without a
re-walk in the supervised case (
cascading_runtime_corruption_cascadesassertsthe machine stays in Ready). Both paths still agree on policy through
gate_by_policy; they differ only in what follows it. Re-walk is recoverysemantics.
Note
gating_the_last_ungated_component_ends_the_walk: a fully gated chainlands in Ready with everything held in reset and no lockdown. That matches the
existing "a non-required cascade never enters recovery or lockdown" rule, but
say so if you read it differently.
Review follow-up: the same in-flight-verdict race was live in
AwaitingReady,reported by @rusty1968.
handle_corruption_advancingis now shared by bothstates that release off
chain[cursor]. It is not called fromhandle_supervising, becauseRecovering's cursor is stale and advancingthere resumes the walk mid-recovery.
Chasing that finding as a class rather than a case turned up two more, both
older than this branch. A
VerificationFailedin flight when the cascade gatedthe component took an isolated component into recovery. A
CorruptionDetectedfor a component the cascade already held did the same, and on retry exhaustion
gate_by_policyread its ownRequiredpolicy and locked the platform down,which contradicts the containment note above. Both are
is_gatedguards.property_isolation_is_sticky_under_random_sequencesis what found them: afterReportIsolated(id), nothing releases that component or hands it to recovery.Verify-before-release caught the two cursor ones and misses the recovery half,
since recovery re-verifies before releasing. The release property now also fuzzes the chain
shape; on the old fixed chain the
AwaitingReadyrace was unreachable.Separately, in
cascade_holditself: it only queued a dependent it had justgated, so the walk stopped at a component isolated earlier and never reached
what depends on it. A component whose whole dependency chain is isolated came
out of reset. The frontier is the visited set now, which still bounds the walk
at one visit per component and terminates on a cycle.
property_cascading_isolation_reaches_the_whole_subtreeguards it, scoped tocascading roots because
Isolabledependents are meant to keep running.