🧹 feat: Retire Stale Linked Worktrees Automatically - #299
Conversation
Agents create one linked worktree per task under <checkout>/.worktrees and nothing ever removed them. Each keeps its own dependencies and build output, so a trusted-VM checkout accumulated 231 worktrees and filled its disk to 100% three times, quarantining workspaces. A worker with linked worktree lanes now retires stale worktrees in the background: two minutes after start, every six hours, sooner while a backlog remains, and before managed setup would be deferred for low disk space. A worktree is retired only when it verifies as a linked worktree under .worktrees, no request uses its lane or checkout, it has been idle for seven days by both this worker's lane use and on-disk Git activity, it is clean, unlocked, outside any merge/rebase/cherry-pick/revert/bisect, unquarantined, and its work is published or merged: HEAD is contained in a remote-tracking ref, or, with no live upstream (detached, never pushed, or upstream deleted after merge), its commits are already in the remote's default branch by content (patch-equivalent per git cherry, or identical content at every path the branch changed). Squash-merged branches and abandoned review worktrees therefore qualify; only local Git is consulted. Removal is `git worktree remove` without --force followed by `git worktree prune`; branches are kept so `git worktree add` restores any worktree. Retirement is on by default. --no-worktree-retirement or LIBRECHAT_CODE_WORKTREE_RETIREMENT=false disables it, and --worktree-idle-days / LIBRECHAT_CODE_WORKTREE_IDLE_DAYS change the threshold. LIBRECHAT_CODE_LOG_LEVEL=debug logs every kept worktree with its reason.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d48e1c7f02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return await this.tracked( | ||
| this.checkoutRequests, | ||
| request.workspaceId, | ||
| () => this.options.delegate.execute(request, signal), | ||
| false, |
There was a problem hiding this comment.
When a checkout request starts after whileIdle performs its one-time checkoutRequests check, this path admits it without waiting for the retirement promise. Root requests can access .worktrees/* (as noted in worker.ts), so such a request can read or write the lane concurrently with the potentially long git worktree remove; a write that begins after Git's cleanliness check can be deleted, or the command can fail as its files disappear. Checkout requests need the same family-level fence used for arriving lane requests.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 82fa757. Valid finding. LinkedWorktreeWorkspaceTools now records each retirement with the workspace ID it belongs to. Checkout requests (execute without a worktree, and programmatic requests without workspace_worktree) now wait for any retirement in progress beneath their checkout before reaching the delegate, the same fence that arriving lane requests already used. whileIdle still refuses to start while a checkout request is in flight. Test: "a lane request that arrives during retirement waits, then finds the lane gone" now also issues a checkout read_file mid-removal, asserts it does not settle until the removal finishes, and then checks that it succeeds.
| for (const checkout of retiredCheckouts) { | ||
| try { | ||
| await git(checkout, ['worktree', 'prune']); |
There was a problem hiding this comment.
After any managed lane is retired, this repository-wide command can also expire metadata for every other missing registered worktree, including worktrees outside .worktrees whose paths are temporarily unavailable. git worktree prune -h describes the operation as expiring working trees older than the configured time; it is not scoped to the lane just removed. Since successful git worktree remove already deletes that lane's metadata, this extra prune can unexpectedly deregister and break an old external worktree.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 82fa757. Valid finding. The repository-wide git worktree prune is gone. git worktree remove already deletes the retired worktree's own .git/worktrees/<name> metadata, so the prune only ever affected other registered worktrees, including ones that are just temporarily unavailable. New test: "retirement never prunes other registered worktrees that are temporarily missing". It moves an external worktree away, retires a lane, and asserts that the external worktree's metadata survives while the retired lane's metadata is gone. The README and PR description now say so explicitly.
| for await (const entry of directory) { | ||
| if (names.length >= SCAN_LIMIT) return { names, truncated: true }; | ||
| if (isValidLinkedWorktreeName(entry.name)) names.push(entry.name); |
There was a problem hiding this comment.
When a checkout contains more than 4,096 entries under .worktrees, every pass reopens the directory from the beginning and returns after the same first batch. The returned truncated flag is only logged and no directory cursor or rotation state advances past these entries, so valid lanes later in the enumeration can remain permanently uninspected and never be reclaimed, precisely for the largest storage-pressure cases.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 82fa757. The per-checkout directory bound is now 65,536 entries, a memory bound far above any real checkout (the production case is ~230), so every pass sees every entry. The per-pass bound that matters is Git inspection: at most 128 idle worktrees inspected and 32 removed. That is already fair across passes through the rotation set added earlier, which inspects not-yet-visited candidates first and keeps scheduling 5-minute continuation passes until a rotation completes. I chose this over a persistent directory cursor because readdir order is not stable across entry creation and deletion, so a cursor could skip entries.
| signal?.throwIfAborted(); | ||
| try { | ||
| // Never --force: Git re-checks for changes and untracked files itself and refuses a dirty or locked worktree. | ||
| await git(lane.checkoutRoot, ['worktree', 'remove', lane.root], undefined, GIT_REMOVE_TIMEOUT_MS); |
There was a problem hiding this comment.
When deleting a large worktree or operating on a slow filesystem takes more than 30 minutes, this timeout causes execFile to terminate Git even though removal is already in progress. The catch then reports an ordinary failed result, but Git may already have deleted the lane's .git file and part of its contents, leaving a half-removed worktree that subsequent passes cannot verify or retire—the exact state the comment above GIT_REMOVE_TIMEOUT_MS says must be avoided.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 82fa757. Valid finding: the 30-minute timeout contradicted the comment above it. git worktree remove now runs with no timeout and no abort signal, so neither a slow filesystem nor worker shutdown can interrupt it mid-delete. WorktreeRetirementScheduler.stop() already awaits the running pass, and the pass checks the abort signal only between worktrees, so shutdown waits for the current removal to finish rather than killing it. The probes before removal (status, rev-parse, for-each-ref, cherry, diff) keep their 30-second timeouts.
| return await this.tracked(this.laneRequests, internalId, async () => { | ||
| await this.retiring.get(internalId); | ||
| return await task(); |
There was a problem hiding this comment.
When a lane request arrives during a long retirement and its caller subsequently aborts, this unconditional await does not observe the request's AbortSignal. The request remains counted as in flight and occupies its assignment or lease until removal settles—potentially for many minutes or indefinitely if removal is allowed to finish safely—and only then invokes the task with an already-aborted signal. The retirement wait should race or otherwise subscribe to cancellation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 82fa757. Valid finding. The wait on a retirement (for lane and checkout requests alike) now races the request's AbortSignal. A cancelled request rejects with the signal's reason as soon as it aborts, leaves the in-flight count through the existing finally, and never runs its task. Test: the same mid-retirement test starts a lane request with an AbortController, aborts it while the removal is gated, and asserts that it rejects with the abort reason before the retirement is released.
…Short - Checkout requests wait for an in-progress retirement of any lane beneath them, since root commands can reach .worktrees/*. - Requests waiting on a retirement honor their own cancellation. - git worktree remove runs without a timeout; a killed removal could leave a half-deleted worktree that no later pass can verify. - Drop the repository-wide git worktree prune: remove already deletes the retired worktree's metadata, and prune would also expire other registered worktrees that are only temporarily unavailable. - Read every .worktrees entry each pass instead of the first 4096.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82fa757a28
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const retire = (): Promise<Kept | 'retired'> => remove(candidate, verdict.head, options.activity, signal); | ||
| const reserved = options.activity | ||
| ? await options.activity.whileIdle(source.workspaceId, name, retire) | ||
| : { ran: true as const, value: await retire() }; |
There was a problem hiding this comment.
If a checkout-level request starts after inspect() checks quarantine, fails ambiguously, and finishes before whileIdle() runs, it can leave the checkout quarantined while no longer appearing in checkoutRequests. This path then removes the lane without rereading either checkout or lane quarantine; because checkout commands can reach .worktrees/* and Git permits removal of ignored files, it can delete state from the uncertain mutation that quarantine was meant to preserve. Recheck both quarantine files inside the reserved retire callback before calling remove().
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in bf294ed. Valid finding. remove(), which runs inside the whileIdle reservation, now re-reads both the checkout's and the lane's quarantine markers before anything else touches the tree. If either is present it keeps the worktree as quarantined, and an unreadable marker also counts as quarantined. Because the reservation excludes every lane and checkout request from that point until removal finishes, no new quarantine can appear between this check and git worktree remove. New test: "a quarantine that appears after inspection is caught under the lane reservation" makes the checkout marker, and then the lane marker, appear only after inspection, and asserts that the worktree is kept both times.
A checkout or lane request that finished between inspection and the lane reservation may have left an uncertain mutation quarantined. Re-read both quarantine markers inside the reservation, immediately before removal.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf294eddf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const retire = (): Promise<Kept | 'retired'> => remove(candidate, verdict.head, options, signal); | ||
| const reserved = options.activity | ||
| ? await options.activity.whileIdle(source.workspaceId, name, retire) | ||
| : { ran: true as const, value: await retire() }; |
There was a problem hiding this comment.
When a successful checkout-level command runs after git status has returned but before this reservation is installed, it can write an existing ignored file—or a file below an existing ignored directory—in .worktrees/<name> and finish. Such a write changes neither the tracked lane activity nor the worktree root directory mtime, checkout requests do not update the lane's lastUsed, and a successful command leaves no quarantine; the final git worktree remove then treats the lane as clean and deletes the newly written ignored data. The new under-reservation quarantine recheck only closes the ambiguous-failure path, so checkout activity must be fenced throughout inspection or recorded and rechecked before removal.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1563f37 with the record-and-recheck option, not by fencing checkout requests during inspection, so a pass never holds back the checkout. LinkedWorktreeWorkspaceTools now exposes checkoutActivity(workspaceId), a counter bumped when each checkout request (outside any lane) starts and when it finishes. Counting both start and finish matters: a request already in flight when inspection begins and finishing before the reservation also changes the counter. The pass captures the counter just before inspecting each worktree. remove() re-reads it under the whileIdle reservation and keeps the worktree as changed for a later pass if any checkout request started or finished in between. From the reservation until removal finishes, checkout requests are fenced as before, so the window is closed end to end. New test: "a checkout request that runs during inspection defers retirement" covers the counter itself, the deferral, and retirement on the next quiet pass.
A checkout request can write ignored files inside .worktrees/<name> that git status never reports, and finish before the lane reservation. Count checkout request starts and finishes, capture the count before inspecting each worktree, and keep the worktree for a later pass when it changed by the time the reservation is held.
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…faultBranch On CI the bare remote's HEAD names master, so the hosting clone had an empty HEAD and `git merge --squash` refused to run.
1 participant
Summary
Agents give every task or pull request its own linked worktree at
<checkout>/.worktrees/<name>, and nothing ever retired them. Each worktree keeps its ownnode_modulesand build output. On the trusted "Skynet" VM the LibreChat checkout reached 231 worktrees, and the disk filled to 100% three times. Each time, workspaces were quarantined because setup and commands could not settle.Two groups dominate. Squash-merged pull requests whose remote branch was deleted: their commits are in
mainby content but on no remote-tracking ref. PR Reviewer subagents create a fresh review worktree for every pass (for examplereview-16677-1f35df8) with its ownnode_modulesand build output, and their best-effort cleanup sometimes fails ("Git reported busy worktree"). Idle, abandoned review worktrees are therefore a major retirement target. Tonight the disk filled to 100% again: 2 MB free at 01:00 UTC, with four lanes quarantined.With this change, a worker running
--linked-worktree-lanesretires stale linked worktrees on its own, on by default and with no configuration.When it runs
storage.minFreeBytes+setupReserveBytesfloor. Space is then measured once more before setup gives up..worktreesentry, inspects at most 128 idle worktrees and removes at most 32, oldest first. A rotation makes sure worktrees kept for a lasting reason, such as unpushed work, cannot hide the rest of a larger backlog. Git probes time out after 30 seconds.git worktree removehas no timeout and ignores shutdown, because a half-deleted worktree could no longer be recognized. One worktree failing never stops the pass.Safety rules
A worktree is retired only when all of these hold. Otherwise it is kept, and the reason is counted and logged at debug level:
It verifies as a linked worktree of a writable registered checkout under
.worktrees/, using the sameverifyLinkedWorktreecheck as lane admission. The main checkout, worktrees outside.worktrees/, and forged directories are never touched.No request for its lane or its checkout is in flight in this worker. It has also been idle longer than the threshold, 7 days by default, measured as the newer of this process's last lane use and on-disk activity: the worktree directory plus its Git
HEAD,index,logs/HEAD,ORIG_HEADandFETCH_HEAD. Removal happens under a lane reservation. A lane or checkout request that arrives meanwhile waits for it, and a lane request then fails as an unknown worktree. Under that reservation, lane use, checkout requests that ran during inspection, quarantine markers, on-disk activity,HEAD, lock and directory identity are all checked again before removal.It has no modified tracked files and no untracked, non-ignored files. No merge, rebase, cherry-pick, revert or bisect is in progress. It is not
git worktree locked, and neither the lane nor its checkout carries a quarantine marker.Its work is published or merged. Either its
HEADcommit, including a detached one, is contained in at least onerefs/remotes/*ref, or its work is already in the remote's default branch by content. The content rule applies only whenHEADhas no live upstream: it is detached, was never pushed, or its upstream ref is gone (deleted after merge). It then also requires one of:git cherry <default> HEADshows only-, and none is a merge commit), which covers rebase merges;The default branch is
refs/remotes/<remote>/HEAD, elsemainormaster, as last fetched. Only local Git is used; the worker never fetches or calls GitHub. Commits beyond a live upstream are never treated as merged.Removal is
git worktree remove <path>without--force, so Git refuses a dirty tree itself. It also deletes that worktree's own.git/worktrees/<name>metadata. A repository-widegit worktree pruneis deliberately not run, because it would also expire other registered worktrees whose paths are only temporarily unavailable. Branches are not deleted, sogit worktree add .worktrees/<name> <branch>restores any retired worktree.Ignored files such as
node_modules, build output and ignored.envfiles go with the worktree; that is the space being reclaimed. A branch that was not merged and whose remote branch was deleted is still kept, because its content is not in the default branch. Host-side Git runs withcore.fsmonitor=false,--no-optional-locks, and no system or global config, like the existing project probes.Defaults and opt-out
--no-worktree-retirementorLIBRECHAT_CODE_WORKTREE_RETIREMENT=false--worktree-idle-days <n>orLIBRECHAT_CODE_WORKTREE_IDLE_DAYS=<n>(1-3650)LIBRECHAT_CODE_LOG_LEVEL=debugInvalid values fail startup. Each pass logs one line, for example
librechat-code: worktree retirement: retired 3, kept 12 (dirty 2, recent 8, unpushed 2), freed about 4.1 GiB. The freed figure is the gain in free space on the checkouts' filesystems, so other writers make it approximate.Changes
worktree-retirement.ts: the retirement pass, the background scheduler and settings parsing.linked-worktrees.ts:LinkedWorktreeWorkspaceToolsnow tracks in-flight lane and checkout requests and last lane use. It exposeslastUsedandwhileIdle, which reserves a lane during removal.snapshot-lifecycle.ts/environment-preparation.ts: the storage guard accepts areclaimSpacehook and measures again once before deferring setup.cli.ts: wires the scheduler, opt-out, threshold, quarantine lookups and logging.Tests
Every test uses real Git repositories with a bare
originremote in temp directories:git worktree add, and the main checkout is untouched..worktrees/<x>and a plain directory.mainbut sits beyond its live upstream. All branches are kept..worktrees/.INVALID_REQUEST.npx tsc --noEmitis clean. In WSL with Node 24.16.0,npm testgives 756 tests: 725 pass, 22 are skipped and 9 fail. All 9 failures also fail on unmodifiedmain(a9528bc) on the same host, which fails 16 there: chmod/ACL storage and programmatic-watchdog tests that need the CI environment. The live SRT lane tests (linked-worktrees-live,root-git-denies-live) pass withLIBRECHAT_CODE_LIVE_SRT_TESTS=1.