Skip to content

fix(server): defer reconciliation during sandbox lifecycle operations - #4983

Open
TaylorMutch wants to merge 1 commit into
mainfrom
fix/4981-reconcile-restart-race/TaylorMutch
Open

TaylorMutch wants to merge 1 commit into
mainfrom
fix/4981-reconcile-restart-race/TaylorMutch

Conversation

@TaylorMutch

Copy link
Copy Markdown
Collaborator

Summary

Prevent reconciliation from failing a sandbox restart with the previous runtime's ContainerExited observation. Release Dev's Docker SSH identity test exposed this race when a sweep observed the stopped container after the gateway had already committed Starting.

Related Issue

Closes #4981

Changes

  • Acquire the existing per-sandbox lifecycle gate without waiting before refreshing and applying a reconciliation observation. Busy sandboxes are deferred to a later sweep.
  • Skip durably pending driver operations, including operations owned by another gateway replica.
  • Add deterministic coverage for local and separate-replica observers, unrelated sandbox progress, successful start completion, and genuine later exits.
  • Document reconciliation behavior in the server README.

Release Notes

Testing

  • Checks appropriate to the affected code and behavior pass: server Clippy with warnings denied, Rust formatting, license headers, Markdown lint, and diff whitespace checks.
  • Unit tests added/updated: regression fails before the fix; all 275 compute tests pass with the fix.
  • E2E tests added/updated: existing Docker SSH host identity test passes against the fixed gateway and published runtime images from 2c31ac7e606c4e175288cee3656b23192a1a36b0, using a freshly built workload fixture. A temporary wrapper routes containers through host.docker.internal for Docker Desktop; repository test code is unchanged.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated: server README

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Oct 27, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 06561f9. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: reconciliation can fail a Docker sandbox during restart

1 participant