🧹 feat: Retire Stale Linked Worktrees Automatically

docs/remote-bridge/worker-runbook.md

@@ -521,6 +521,35 @@ replace the local clear command.
 Never clear quarantine merely to make the worker start. It represents a setup,
 command, cancellation, or settlement whose effects may be incomplete.
 
+### Disk filled by task worktrees
+
+Agents that create one linked worktree per task under `<checkout>/.worktrees/`
+leave each one behind with its own dependencies and build output. A worker with
+`--linked-worktree-lanes` retires stale ones automatically: two minutes after
+startup, every six hours, and before managed setup would be deferred for low
+space. Only verified linked worktrees in writable roots that are idle for seven
+days, clean, unlocked, outside any merge or rebase, unquarantined, and whose
+`HEAD` is on a remote-tracking ref or already in the default branch by content
+(a squash- or rebase-merged branch whose remote branch was deleted) are
+removed, with `git worktree remove` without `--force`. Merged content is judged
+against the checkout's last fetch of the default branch; the worker never
+fetches. Branches are kept; restore one with
+`git worktree add .worktrees/<name> <branch>`. The
+[worker package guide](../../packages/code/README.md#retiring-stale-linked-worktrees)
+lists every condition.
+
+Each pass logs `worktree retirement: retired N, kept M (reason counts)`. To see
+why a particular worktree is kept, restart once with
+`LIBRECHAT_CODE_LOG_LEVEL=debug`. Worktrees kept as `dirty` or `unpushed` hold
+work nobody has published; push or discard it from the checkout before removing
+them by hand, never with `rm -rf`. If the disk is already full and setup keeps
+failing, free space by hand first: retirement runs only after the worker starts,
+or during setup that has a `storage` floor configured.
+
+To keep every worktree, add `--no-worktree-retirement` (or
+`LIBRECHAT_CODE_WORKTREE_RETIREMENT=false`). To keep them longer, set
+`--worktree-idle-days <n>` (or `LIBRECHAT_CODE_WORKTREE_IDLE_DAYS`).
+
 ## 15. Common failures
 
 -   **`--environment cannot be combined...`:** remove old workspace flags and
@@ -540,8 +569,13 @@ command, cancellation, or settlement whose effects may be incomplete.
     configure private copy-on-write snapshots and their lifecycle budget. Do not
     symlink another branch's mutable `node_modules` or hardlink writable installs.
 -   **Managed preparation deferred for low space:** `storage.minFreeBytes` plus
-    `setupReserveBytes` is a soft pre-setup floor, not a hard quota. Expand the
-    volume or clean reproducible artifacts; do not clear quarantine as a disk fix.
+    `setupReserveBytes` is a soft pre-setup floor, not a hard quota. A worker
+    with linked worktree lanes first retires stale task worktrees and measures
+    again. If space is still short, expand the volume or clean reproducible
+    artifacts; do not clear quarantine as a disk fix.
+-   **Many `.worktrees/*` directories remain:** see
+    [Disk filled by task worktrees](#disk-filled-by-task-worktrees). Count the
+    kept reasons in the retirement summary before removing anything by hand.
 -   **Snapshot maintenance:** preview with `prune-environment-storage
     --environment <file>` and use `--apply` only after reviewing its JSON. Active,
     unknown and unmarked data stays intact. Include all environment definitions

packages/code/README.md

@@ -948,6 +948,63 @@ be combined with conversation worktrees. Code API must advertise
 scope for them. Deploy consumers that read worker status (such as LibreChat)
 with support for `workspaceScopes` before enabling lanes on a worker.
 
+##### Retiring stale linked worktrees
+
+Nothing else removes a task's worktree once its work is pushed, and each one
+keeps its own dependencies and build output. A worker with lanes therefore
+retires stale ones itself, with no configuration: a pass runs two minutes after
+startup and every six hours after that, sooner while a backlog remains, and
+also before managed environment setup would be deferred for low disk space.
+Passes run in the background and never overlap. Only an actual removal holds
+back requests, and only those for that lane or its checkout.
+
+A worktree in a writable registered root is retired only when **all** of these
+hold; otherwise it is kept and the reason is counted:
+
+- It verifies as a linked worktree of the checkout under `.worktrees/`, as for
+  lane admission. The checkout itself and worktrees elsewhere are never
+  touched.
+- Neither the lane nor its checkout has a request in flight or ran one while
+  the worktree was being inspected, and both this
+  worker's last use of the lane and the newest on-disk activity (the worktree
+  directory and its Git `HEAD`, `index`, `logs/HEAD`, `ORIG_HEAD` and
+  `FETCH_HEAD`) are older than the idle threshold, seven days by default.
+- It has no modified tracked files and no untracked files the repository does
+  not ignore; no merge, rebase, cherry-pick, revert or bisect in progress; no
+  `git worktree lock`; and no quarantine on the lane or its checkout.
+- Its `HEAD` commit, including a detached one, is contained in at least one
+  remote-tracking ref (`refs/remotes/*`), or its work is already in the
+  remote's default branch by content, as after a squash or rebase merge whose
+  remote branch was deleted. Content counts only when `HEAD` has no live
+  upstream (it is detached, never pushed, or its upstream ref is gone) and
+  either every commit beyond the default branch is patch-equivalent to one in
+  it (`git cherry` shows only `-`, with no merge commits), or the default
+  branch has identical content at every path the branch changed since their
+  merge base. The default branch is the remote's `HEAD`, else `main` or
+  `master`, as last fetched; the worker never fetches or calls a hosting API.
+  Commits beyond a live upstream are never treated as merged.
+
+Removal is `git worktree remove` **without** `--force`, so Git re-checks for
+changes itself and deletes only that worktree's metadata. No repository-wide
+`git worktree prune` runs, so other registered worktrees that are temporarily
+unavailable stay registered. Removal has no timeout, because a half-deleted
+worktree could no longer be recognized. Ignored files such as `node_modules`,
+build output and ignored `.env` files go with the worktree; that is the space
+being reclaimed. The branch is kept, so
+`git worktree add .worktrees/<name> <branch>` restores the worktree. Lane and
+checkout requests that arrive during a removal wait for it, or for their own
+cancellation; a lane request then fails as an unknown worktree. Each pass reads
+every `.worktrees` entry, inspects at most 128 idle worktrees and removes at
+most 32, oldest first, rotating so that worktrees kept for lasting reasons
+cannot hide the rest. Each pass logs one summary line, for example
+`worktree retirement: retired 3, kept 12 (dirty 2, recent 8, unpushed 2), freed
+about 4.1 GiB`; set `LIBRECHAT_CODE_LOG_LEVEL=debug` to log every kept worktree
+and its reason.
+
+Pass `--no-worktree-retirement` or set `LIBRECHAT_CODE_WORKTREE_RETIREMENT=false`
+to disable retirement. Change the idle threshold with `--worktree-idle-days <n>`
+or `LIBRECHAT_CODE_WORKTREE_IDLE_DAYS=<n>` (1 to 3650 days).
+
 On an updated Code API, admission waits up to 30 seconds without the
 `X-LibreChat-Workspace-Queue-Wait-Ms` request header. A caller may advertise a
 positive integer millisecond allowance up to five minutes, capped by any server

packages/code/src/cli.ts

@@ -43,6 +43,10 @@ import { NativeWorkspaceCommandPool } from './native-pool.js';
 import { GitWorktreeWorkspaceTools, internalWorkspaceId } from './workspace-instances.js';
 import { LINKED_WORKTREE_DIRECTORY, LinkedWorktreeWorkspaceTools } from './linked-worktrees.js';
 import { GitWorktreeManager } from './worktrees.js';
+import {
+  WorktreeRetirementScheduler,
+  worktreeRetirementSettings,
+} from './worktree-retirement.js';
 import { captureWorkspaceRootIdentity } from './root-identity.js';
 import {
   resolveNativeSrtCommandPolicy,
@@ -812,6 +816,13 @@ async function run(
   if (linkedWorktreeLanes && process.platform === 'win32') {
     throw new Error('Linked worktree Git guard requires a POSIX host');
   }
+  const worktreeRetirement = worktreeRetirementSettings({
+    optOut: args.includes('--no-worktree-retirement'),
+    enabled: process.env.LIBRECHAT_CODE_WORKTREE_RETIREMENT,
+    idleDays:
+      option(args, '--worktree-idle-days') ??
+      process.env.LIBRECHAT_CODE_WORKTREE_IDLE_DAYS,
+  });
   if (
     roots.length > 1 &&
     process.env.LIBRECHAT_CODE_WORKSPACE_QUARANTINE_FILE?.trim()
@@ -1301,6 +1312,41 @@ async function run(
     });
     workspaceTools = linkedWorktreeTools;
   }
+  const retirementSources = roots.filter((root) => root.writable);
+  const debugLogs =
+    process.env.LIBRECHAT_CODE_LOG_LEVEL?.trim().toLowerCase() === 'debug';
+  const worktreeRetirementScheduler =
+    linkedWorktreeTools &&
+    worktreeRetirement.enabled &&
+    retirementSources.length > 0 &&
+    option(args, '--reset-workspace-quarantine') == null
+      ? new WorktreeRetirementScheduler({
+          activity: linkedWorktreeTools,
+          idleMs: worktreeRetirement.idleMs,
+          sources: retirementSources.map((root) => ({
+            workspaceId: root.id,
+            root: root.root,
+            identity: root.identity,
+          })),
+          async isQuarantined(selectedWorkspaceId, worktree) {
+            const source = roots.find((root) => root.id === selectedWorkspaceId);
+            if (!source) return true;
+            const path =
+              worktree == null
+                ? rootQuarantinePaths.get(selectedWorkspaceId)!
+                : defaultWorkspaceQuarantinePath({
+                    codeApiUrl,
+                    workerId,
+                    workspaceRoot: join(source.root, LINKED_WORKTREE_DIRECTORY, worktree),
+                  });
+            return (await loadWorkspaceMutationQuarantine(path)) != null;
+          },
+          log(level, message) {
+            if (level === 'debug' && !debugLogs) return;
+            process.stdout.write(`librechat-code: ${message}\n`);
+          },
+        })
+      : undefined;
     if (workspaceTools && environments.length) {
         workspaceTools = new EnvironmentWorkspaceTools(
             workspaceTools,
@@ -1375,6 +1421,9 @@ async function run(
                 root: environment.definition.root,
                 identity: roots.find(root => root.id === id)!.identity!,
                 setup, receiptPath: preparationReceipt(environment.definition.root),
+                reclaimSpace: worktreeRetirementScheduler
+                  ? () => worktreeRetirementScheduler.runNow()
+                  : undefined,
                 context: JSON.stringify([serializeNativeSrtCommandPolicy(commandPolicy), commandAllowedDomains, github.policyIdentity,
                   nativeOptionsForWorkspace(id).resources]),
                 signal: controller.signal,
@@ -1404,10 +1453,12 @@ async function run(
             );
         }
   } catch (error) {
+    await worktreeRetirementScheduler?.stop().catch(() => undefined);
     await nativeCommandSandbox?.close().catch(() => undefined);
     await fileRelaySupervisor?.stop().catch(() => undefined);
     throw error;
   }
+  worktreeRetirementScheduler?.start();
   try {
     const worker = new BridgeWorker({
       instructionDescriptors: () => localWorkspaceTools?.instructionDescriptors() ?? Promise.resolve(undefined),
@@ -1617,6 +1668,7 @@ async function run(
     await worker.run(controller.signal);
   } finally {
     try {
+      await worktreeRetirementScheduler?.stop();
       await nativeCommandSandbox?.close();
     } finally {
       await fileRelaySupervisor?.stop();

packages/code/src/environment-preparation.ts

@@ -37,6 +37,8 @@ export interface EnvironmentPreparationOptions {
     snapshotScope?: string;
     beforeMutation?(): Promise<void>;
     storage?: EnvironmentStoragePolicy;
+    /** Frees reproducible storage when the storage floor would defer setup. */
+    reclaimSpace?(): Promise<unknown>;
 }
 
 /** Checkout-local reuse. Never transfers mutable installations between worktrees. */
@@ -87,7 +89,12 @@ async function prepareInLock(
     }
     if (options.snapshotStore && portable) {
         if (options.storage)
-            await assertPreparationSpace(options.root, options.storage);
+            await assertPreparationSpace(
+                options.root,
+                options.storage,
+                undefined,
+                options.reclaimSpace,
+            );
         await options.beforeMutation?.();
         if (
             await restoreDependencySnapshot(
@@ -118,7 +125,12 @@ async function prepareInLock(
         }
     }
     if (options.storage)
-        await assertPreparationSpace(options.root, options.storage);
+        await assertPreparationSpace(
+            options.root,
+            options.storage,
+            undefined,
+            options.reclaimSpace,
+        );
     const result = await options.execute(
         options.setup.command,
         options.setup.timeoutMs,