Skip to content

orchestrator: Move the cursor off a component gated mid-walk - #454

Merged
chrysh merged 3 commits into
OpenPRoT:mainfrom
9elements:fix-midwalk-gated-cursor
Sep 16, 2026
Merged

chrysh merged 3 commits into
OpenPRoT:mainfrom
9elements:fix-midwalk-gated-cursor

Conversation

@chrysh

@chrysh chrysh commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Two things, found by asking whether a cascade that fires mid-walk is covered.

The test: mid_walk_cascade_skips_unreached_dependents. The existing cascade
tests all corrupt a component after the walk reaches Ready, so the cursor is
already past everything the cascade gates, and the is_gated skip in
advance_to_next_ungated is never exercised at depth. The new test corrupts C1
while C0 is still under verification: C1 -> C2 -> C3 are gated before their turn
and none is ever read or verified.

The fix: writing that test showed that gating the component under verification
left the cursor on it, and release keys off chain[cursor] alone, so a
VerificationPassed already in flight took an isolated component back out of
reset. PreSupervision's CorruptionDetected arm now advances the cursor past
a gated component (Ready if the rest of the chain is gated). The late verdict is
then a cursor mismatch and is dropped, and the walk does not wait on a verdict it
must ignore.

Why advance in place rather than re-enter PreSupervision the way the
recovery-exhaustion path does: corruption already gates and continues without a
re-walk in the supervised case (cascading_runtime_corruption_cascades asserts
the machine stays in Ready). Both paths still agree on policy through
gate_by_policy; they differ only in what follows it. Re-walk is recovery
semantics.

Note gating_the_last_ungated_component_ends_the_walk: a fully gated chain
lands in Ready with everything held in reset and no lockdown. That matches the
existing "a non-required cascade never enters recovery or lockdown" rule, but
say so if you read it differently.

Review follow-up: the same in-flight-verdict race was live in AwaitingReady,
reported by @rusty1968. handle_corruption_advancing is now shared by both
states that release off chain[cursor]. It is not called from
handle_supervising, because Recovering's cursor is stale and advancing
there resumes the walk mid-recovery.

Chasing that finding as a class rather than a case turned up two more, both
older than this branch. A VerificationFailed in flight when the cascade gated
the component took an isolated component into recovery. A CorruptionDetected
for a component the cascade already held did the same, and on retry exhaustion
gate_by_policy read its own Required policy and locked the platform down,
which contradicts the containment note above. Both are is_gated guards.

property_isolation_is_sticky_under_random_sequences is what found them: after
ReportIsolated(id), nothing releases that component or hands it to recovery.
Verify-before-release caught the two cursor ones and misses the recovery half,
since recovery re-verifies before releasing. The release property now also fuzzes the chain
shape; on the old fixed chain the AwaitingReady race was unreachable.

Separately, in cascade_hold itself: it only queued a dependent it had just
gated, so the walk stopped at a component isolated earlier and never reached
what depends on it. A component whose whole dependency chain is isolated came
out of reset. The frontier is the visited set now, which still bounds the walk
at one visit per component and terminates on a cycle.
property_cascading_isolation_reaches_the_whole_subtree guards it, scoped to
cascading roots because Isolable dependents are meant to keep running.

@chrysh
chrysh force-pushed the fix-midwalk-gated-cursor branch from 59705ad to 8db5243 Compare August 31, 2026 20:09
@chrysh
chrysh marked this pull request as ready for review August 31, 2026 20:09

@embediver embediver left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests look all good.
On the actual fix I'm not sure if I understood it completely but looks good otherwise.

@rusty1968 rusty1968 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness (blocking)

The cursor-advance fix is applied only in PreSupervision's CorruptionDetected arm. The identical race is left open in AwaitingReady, which handles CorruptionDetected through handle_supervising's bare self.handle_corruption(*id, ctx) (lib.rs:773) — this call site was not touched by the diff.

AwaitingReady has the same chain[cursor]-keyed VerificationPassed release logic that motivated this fix in PreSupervision (lib.rs:598-621), and its cursor points at a real, currently-under-verification component by construction (that's exactly how a machine enters AwaitingReady — see lib.rs:517-528). Concretely: chain [C0: active_required, C1: passive_cascading, C2: passive_required]. BOOT; VerificationPassed(C0)AwaitingReady(Some(C0)), cursor on C1 (VerifyFirmware(C1) already emitted). CorruptionDetected(C1) falls through AwaitingReady's match (no arm for it) to handle_supervising, which gates C1 via handle_corruption but never advances the cursor off it. A VerificationPassed(C1) already in flight — the same in-flight-verdict scenario this PR's own comment calls out — then matches chain[cursor] == C1 in AwaitingReady's own VerificationPassed arm and fires Effect::ReleaseReset(C1), taking a component the cascade just isolated and reported (ReportIsolated(C1)) back out of reset. That's the exact bug this PR fixes for PreSupervision, still live one state over.

Fix: factor the new logic in the PreSupervision arm into a shared method (e.g. handle_corruption_advancing) and call it from both the PreSupervision arm and handle_supervising's CorruptionDetected arm. This is a no-op for Ready/Updating (cursor is at the past-the-end sentinel there) and for Recovering (its own handler doesn't key release off chain[cursor]), so it's safe to apply uniformly. See suggested diff, which also adds a regression test mirroring the new PreSupervision tests but for AwaitingReady.

Verbosity/duplication (ponytail-review, non-blocking)

The "advance, then Handled if found else Transition(Ready)" pattern now appears a third time (PreSupervision's new corruption arm, PreSupervision's VerificationPassed, AwaitingReady's VerificationPassed). The suggested fix above naturally collapses one of those copies into the new shared helper; consider whether it's worth doing for all three, but that's optional and not required for this PR.

No issues found in the no-alloc / no-panic / secure-coding categories — the new code uses heapless/saturating_add consistently with the rest of the file, no unwrap/panics, no secret handling involved.

Test coverage for the three new PreSupervision scenarios is solid and follows existing conventions; it just needs a sibling test for the AwaitingReady path described above.

@chrysh

chrysh commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

You were right. Asking why it was only PreSupervision turned up two more like
it.

The fix: AwaitingReady has its own CorruptionDetected arm now, calling a
shared handle_corruption_advancing.
mid_walk_cascade_in_awaiting_ready_drops_late_verdict is your scenario. The
invariant is on the cursor field now: while the walk runs, the cursor never
points at a gated component.

Why its own arm rather than handle_supervising: Recovering reaches that
handler too, and its cursor is stale, so the uniform version resumes the walk
mid-recovery. I tried it, and
corruption_during_recovery_does_not_advance_the_walk is the only one of the 93
tests that fails. Ready and Updating are no-ops as you say, so only
Recovering forces the placement.

The other two are older than this branch. A VerificationFailed in flight when
the cascade gated the component sent an isolated component into recovery. And
handle_corruption ran gate_by_policy on every report, including one for a
component already held, so a cascade-held Required component went to recovery
and the third report exhausted its retries: gate_by_policy then read that same
Required policy and locked the platform down, over a cascade that was
already contained. Both fixes are one-line is_gated guards.

Why a property rather than three more tests: four of these is a class.
property_isolation_is_sticky_under_random_sequences says that after
ReportIsolated(id) nothing releases that component or hands it to recovery. It
finds all four; the release property only ever caught the two cursor ones, and
misses the recovery half because recovery re-verifies before releasing. It also
fuzzes the chain shape now, which is what made your case reachable at all.

One more, in the cascade itself. cascade_hold only queued a dependent it had
just gated, so the walk stopped at a component isolated earlier and never
reached what depends on it: gate C1 on its own, then corrupt cascading C0, and
C2 comes out of reset with both components it depends on isolated. The frontier
is the visited set now, and
property_cascading_isolation_reaches_the_whole_subtree guards it, scoped to
cascading roots since Isolable dependents are meant to keep running. Neither
existing property saw that one: C2 was verified before release, and never
isolated.

The third copy I left alone: the other two are VerificationPassed arms where
the advance is interleaved with release bookkeeping, and PreSupervision's also
branches on ComponentKind::Active. Say so if you want it collapsed anyway.

A cascade can gate the component currently under verification. Only
chain[cursor] can be released, so leaving the cursor on that component let its
in-flight verdict take it back out of reset. handle_corruption_advancing gates
by policy and then moves the cursor to the next ungated component. The two
states that release only chain[cursor], PreSupervision and AwaitingReady, route
CorruptionDetected through it.

It is not called from handle_supervising. Recovering's cursor is stale, since
VerificationFailed leaves it on the failed component, so advancing there would
verify mid-recovery or reach Ready instead of re-walking.

The release property now randomizes the chain shape as well as the event
sequence. The AwaitingReady race needs a gateable component after an active
one, which the old fixed chain never had.

Also adds the missing test for the Chain::try_from length bound. cursor is a u8
using chain.len() as its past-the-end sentinel, so a 256-entry chain would
truncate that sentinel to 0 and the walk would never read as done.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Christina Quast <christina.quast@9elements.com>
A failure verdict or a corruption report can be in flight from before the
cascade gated its component. VerificationFailed and CorruptionDetected now
return early when the component is gated. Recovering it re-walks the chain for
a device that stays held, and on exhaustion a Required one locks the platform
down over a cascade that was already contained, which contradicts the rule that
a non-Required cascade never reaches lockdown.

property_isolation_is_sticky_under_random_sequences guards both: after
ReportIsolated(id), nothing in the rest of the run emits ReleaseReset or
RecoverComponent for that id. The cursor invariant the walk depends on is
written down on the field, since that property is what guards it.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Christina Quast <christina.quast@9elements.com>
cascade_hold only queued a dependent it had just gated, so the walk stopped at a
component isolated earlier and never reached what depends on it. Gate a middle
node on its own, then corrupt its Cascading parent, and the component behind it
comes out of reset with its whole dependency chain isolated.

The frontier is the visited set now, so traversal continues through a gated
component. gate_one is idempotent, so re-visiting one emits nothing.

property_cascading_isolation_reaches_the_whole_subtree guards it.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Christina Quast <christina.quast@9elements.com>
@chrysh
chrysh force-pushed the fix-midwalk-gated-cursor branch from 92ab8df to e19c5e5 Compare September 16, 2026 13:03

@leongross leongross left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@chrysh
chrysh dismissed rusty1968’s stale review September 16, 2026 14:56

Leon approved the PR. Rusty1968's changes were taken into consideration.

@chrysh
chrysh merged commit 7fde1ef into OpenPRoT:main Sep 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants