Skip to content

🤖 fix: session disposal does not wait for a stopping migration - #5592

Merged
ThomasK33 merged 2 commits into
mainfrom
fix/5589-disposal-stuck-migration
Oct 3, 2026
Merged

ThomasK33 merged 2 commits into
mainfrom
fix/5589-disposal-stuck-migration

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Since #5579, a refused or failed background migration whose exit is never confirmed stays pending for the rest of the backend process. Session disposal's cleanup(workspaceId) drains pending entries with no deadline, so it hung forever and kept its admission seal. After that, every later move to the background in that workspace was refused, and session.end and listener removal never ran. Now session disposal does not wait for a migration whose command is being stopped. It finishes and releases its seal. Removal and archive still wait for that migration and fail closed at their 60 s drain deadline.

Fixes #5589

Implementation

  • beginMigration() returns a markStopping(). The bash tool calls it where it hands a refused or failed migration to the command's exit (🤖 fix: keep a refused or failed migration pending until its command exits #5579).
  • The manager remembers stopping entries in a WeakSet beside the existing pending set. No new persisted state.
  • drainPendingAdmissions without a deadline (only session disposal uses that form) skips stopping entries. With a deadline (removal and archive, failClosedAfterDrainTimeout), it waits for every entry as before.
  • Disposal can start while a migration is still on its way to registering, which then fails. Each markStopping() therefore also wakes a disposal drain that is already waiting, and the drain re-checks its entries. Test: "lets session disposal finish once a migration it waits for becomes stopping" (backgroundProcessManager.test.ts). Without the wake it times out with Expected: "finished", Received: "waiting".

Why this rather than a disposal deadline (option a) or a process-group probe (option b)

  • A plain deadline on the disposal drain would also stop waiting for admitted spawns that may still register. The existing cleanup doc forbids that: the spawn could register after disposal lifted its seal, with no cleanup left to stop it. The deadline would also add a timer and an arbitrary wait to every disposal.
  • A stopping migration can never register. So disposal, which deletes nothing, has nothing to wait for. Marking those entries fixes exactly the regression 🤖 fix: keep a refused or failed migration pending until its command exits #5579 introduced, with no new timing. Disposal behaves as before for every other pending entry.
  • A process-group probe (b) would need a probe per runtime, including remote runtimes, which this change does not. It also does not help removal, which must keep failing closed anyway.

Validation

  • Test-first: "a migration whose exit is never confirmed blocks removal but not session disposal" in backgroundProcessesFormalRepro.test.ts. Its fake runtime rejects exitCode after the kill, so the exit is never confirmed. Without the fix, disposal is still waiting after 2 s:

    Expected: "finished"
    Received: "waiting"
    (fail) ... > a migration whose exit is never confirmed blocks removal but not session disposal
    

    With the fix, disposal finishes, a later beginMigration is admitted (the seal is released), and a removal's cleanup(ws, { failClosedAfterDrainTimeout: true }) is still waiting. The test helper's afterEach disposal cleanup no longer needs a special case for this runtime.

  • Bun 1.3.12: the repro file (16 tests), bash*, backgroundProcessManager, workspaceService.archive/remove, serviceContainer and agentSession.disposeRace pass (459 tests). make static-check passes. One run hit an unrelated flake in a non-host spawn test. It passed alone 3/3 here and 2/2 on main, and it is filed as 🤖 flake: two backends' same-name non-host spawns got one record directory under load #5591.

  • Formal: formal/background-processes models removal and archive only, not session disposal, and this PR changes no formal files. Removal and archive behaviour is unchanged.

Risks

Low. The change only affects session disposal while a stopping migration is pending. Disposal now returns while that command may still be stopping. Disposal never deleted anything, and the entry stays tracked for removal and archive.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $44.17

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-03T21:54:06.461790Z 450bb83 New commits
🔒 Security Review ✅ Completed 2026-10-03T21:55:06.120570Z 450bb83 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eee83d782a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/backgroundProcessManager.ts Outdated
@ThomasK33

Copy link
Copy Markdown
Member Author

Readiness record for head 450bb83:

  • CI: all checks pass on this head (29 pass, 4 skipped). The first Codex Comments run saw a review thread that was still unresolved at the time. After I resolved it, I re-ran that job and it passed. Unresolved threads: 0.
  • Local: make static-check passes. 459 targeted tests pass on Bun 1.3.12. One unrelated spawn-test flake is filed as 🤖 flake: two backends' same-name non-host spawns got one record directory under load #5591.
  • Review budget: 5 of 6 assessments. Normal reviews: round 1 had 1 finding (a disposal drain could miss a later markStopping()), fixed with a test, and round 2 had 0. Security: 0 and 0. Findings strictly shrank.
  • Final independent check (clean context, on exactly this head): "ready with tracked follow-ups". No blocker. Its follow-ups are tracked in 🤖 bug: removal stays refused until restart after a migration's exit is never confirmed #5594: a stale failClosedAfterDrainTimeout doc line, and a recovery path for removal after an exit that is never confirmed (fail-closed by design).
  • Decision: ready. Merging with squash, pinned to this head.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $44.17

@ThomasK33
ThomasK33 added this pull request to the merge queue Oct 3, 2026
Merged via the queue into main with commit b714eb6 Oct 3, 2026
60 of 62 checks passed
@ThomasK33
ThomasK33 deleted the fix/5589-disposal-stuck-migration branch October 3, 2026 22:20
yermakoffivan pushed a commit to yermakoffivan/mux that referenced this pull request Oct 4, 2026
…ep 2 (coder#5595)

## Summary

The three foreground-to-background migration tests in
`tests/ipc/runtime/backgroundBashDirect.test.ts` relied on a fixed
`sleep 2`: the command had to keep running for 2 s while the bash tool
claimed a record name and migrated it. On Windows CI that migration can
take longer, so the command finished first and the tool correctly took
the normal completion path (coder#4878). Now each script waits for a release
file that the test creates after it has checked the migration. The test
then waits for the process to exit instead of sleeping 2.5 s. This is a
test-only change.

Fixes coder#5593

## Not a regression from coder#5579 or coder#5592

The same test failed on Windows before coder#5579 (merged 2026-10-03):
- 2026-09-26, run 36272619908 (coder#4740, merge queue), logged on coder#4463.
- 2026-09-27, run 36284062610 (coder#4794), logged on coder#4463. In this one, the
`AFTER_BG` line was already in the output when the migration returned,
which is the same timing problem.

coder#5579 and coder#5592 only change the refused and failed migration paths. The
successful migration path that these tests use is unchanged.

## Root cause

In the merge-queue failure (run 37154303115), `result.success` was true
but `result.backgroundProcessId` was `undefined`
(`backgroundBashDirect.test.ts:502`). So the tool returned a normal
completion. `bash.ts` first awaits the migration's name claim, then
checks whether the command already exited, and if so completes normally.
With a 2 s command, any claim slower than about 2 s produces exactly
this result. Jest reported the failure location as `afterAll`, line 439,
but the failing assertion is the one at line 502.

## Validation

- Reproduced on Linux with a probe (not committed) that delays
`claimMigrationProcessId` by 2.5 s, which matches a slow Windows claim.
On main, all three tests fail with the CI symptom:

  ```text
● Foreground to Background Migration › should migrate foreground bash to
background and continue running
    Expected: "fg_to_bg_…"
    Received: undefined
● … › should preserve output across stream boundaries (Received:
undefined)
● … › should not kill backgrounded process when abort signal fires
(Received: undefined)
  ```

With this change and the same probe, all four tests in that describe
pass.
- Without the probe, the whole file passes (11 tests, `bun x jest`, Bun
1.3.12). `make static-check` passes.
- Windows: the gate path uses forward slashes (`C:/…`), which Git Bash
accepts. The scripts never escape backslashes.

---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high` • Cost: `$47.22`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs=47.22 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🤖 bug: a migration whose exit is never confirmed blocks its workspace's teardown until restart

1 participant