From d48e1c7f02c5682e91b294ccd72d93354ea9b130 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Fri, 2 Oct 2026 21:25:39 -0400 Subject: [PATCH 1/5] =?UTF-8?q?=F0=9F=A7=B9=20feat:=20Retire=20Stale=20Lin?= =?UTF-8?q?ked=20Worktrees=20Automatically?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Agents create one linked worktree per task under /.worktrees and nothing ever removed them. Each keeps its own dependencies and build output, so a trusted-VM checkout accumulated 231 worktrees and filled its disk to 100% three times, quarantining workspaces. A worker with linked worktree lanes now retires stale worktrees in the background: two minutes after start, every six hours, sooner while a backlog remains, and before managed setup would be deferred for low disk space. A worktree is retired only when it verifies as a linked worktree under .worktrees, no request uses its lane or checkout, it has been idle for seven days by both this worker's lane use and on-disk Git activity, it is clean, unlocked, outside any merge/rebase/cherry-pick/revert/bisect, unquarantined, and its work is published or merged: HEAD is contained in a remote-tracking ref, or, with no live upstream (detached, never pushed, or upstream deleted after merge), its commits are already in the remote's default branch by content (patch-equivalent per git cherry, or identical content at every path the branch changed). Squash-merged branches and abandoned review worktrees therefore qualify; only local Git is consulted. Removal is `git worktree remove` without --force followed by `git worktree prune`; branches are kept so `git worktree add` restores any worktree. Retirement is on by default. --no-worktree-retirement or LIBRECHAT_CODE_WORKTREE_RETIREMENT=false disables it, and --worktree-idle-days / LIBRECHAT_CODE_WORKTREE_IDLE_DAYS change the threshold. LIBRECHAT_CODE_LOG_LEVEL=debug logs every kept worktree with its reason. --- docs/remote-bridge/worker-runbook.md | 38 +- packages/code/README.md | 50 ++ packages/code/src/cli.ts | 52 ++ packages/code/src/environment-preparation.ts | 16 +- packages/code/src/linked-worktrees.ts | 154 +++- packages/code/src/snapshot-lifecycle.test.ts | 48 ++ packages/code/src/snapshot-lifecycle.ts | 24 +- packages/code/src/worktree-retirement.test.ts | 520 ++++++++++++++ packages/code/src/worktree-retirement.ts | 679 ++++++++++++++++++ 9 files changed, 1544 insertions(+), 37 deletions(-) create mode 100644 packages/code/src/worktree-retirement.test.ts create mode 100644 packages/code/src/worktree-retirement.ts diff --git a/docs/remote-bridge/worker-runbook.md b/docs/remote-bridge/worker-runbook.md index 51018940..4026a755 100644 --- a/docs/remote-bridge/worker-runbook.md +++ b/docs/remote-bridge/worker-runbook.md @@ -521,6 +521,35 @@ replace the local clear command. Never clear quarantine merely to make the worker start. It represents a setup, command, cancellation, or settlement whose effects may be incomplete. +### Disk filled by task worktrees + +Agents that create one linked worktree per task under `/.worktrees/` +leave each one behind with its own dependencies and build output. A worker with +`--linked-worktree-lanes` retires stale ones automatically: two minutes after +startup, every six hours, and before managed setup would be deferred for low +space. Only verified linked worktrees in writable roots that are idle for seven +days, clean, unlocked, outside any merge or rebase, unquarantined, and whose +`HEAD` is on a remote-tracking ref or already in the default branch by content +(a squash- or rebase-merged branch whose remote branch was deleted) are +removed, with `git worktree remove` without `--force`. Merged content is judged +against the checkout's last fetch of the default branch; the worker never +fetches. Branches are kept; restore one with +`git worktree add .worktrees/ `. The +[worker package guide](../../packages/code/README.md#retiring-stale-linked-worktrees) +lists every condition. + +Each pass logs `worktree retirement: retired N, kept M (reason counts)`. To see +why a particular worktree is kept, restart once with +`LIBRECHAT_CODE_LOG_LEVEL=debug`. Worktrees kept as `dirty` or `unpushed` hold +work nobody has published; push or discard it from the checkout before removing +them by hand, never with `rm -rf`. If the disk is already full and setup keeps +failing, free space by hand first: retirement runs only after the worker starts, +or during setup that has a `storage` floor configured. + +To keep every worktree, add `--no-worktree-retirement` (or +`LIBRECHAT_CODE_WORKTREE_RETIREMENT=false`). To keep them longer, set +`--worktree-idle-days ` (or `LIBRECHAT_CODE_WORKTREE_IDLE_DAYS`). + ## 15. Common failures - **`--environment cannot be combined...`:** remove old workspace flags and @@ -540,8 +569,13 @@ command, cancellation, or settlement whose effects may be incomplete. configure private copy-on-write snapshots and their lifecycle budget. Do not symlink another branch's mutable `node_modules` or hardlink writable installs. - **Managed preparation deferred for low space:** `storage.minFreeBytes` plus - `setupReserveBytes` is a soft pre-setup floor, not a hard quota. Expand the - volume or clean reproducible artifacts; do not clear quarantine as a disk fix. + `setupReserveBytes` is a soft pre-setup floor, not a hard quota. A worker + with linked worktree lanes first retires stale task worktrees and measures + again. If space is still short, expand the volume or clean reproducible + artifacts; do not clear quarantine as a disk fix. +- **Many `.worktrees/*` directories remain:** see + [Disk filled by task worktrees](#disk-filled-by-task-worktrees). Count the + kept reasons in the retirement summary before removing anything by hand. - **Snapshot maintenance:** preview with `prune-environment-storage --environment ` and use `--apply` only after reviewing its JSON. Active, unknown and unmarked data stays intact. Include all environment definitions diff --git a/packages/code/README.md b/packages/code/README.md index 38a22d5d..3928abf7 100644 --- a/packages/code/README.md +++ b/packages/code/README.md @@ -948,6 +948,56 @@ be combined with conversation worktrees. Code API must advertise scope for them. Deploy consumers that read worker status (such as LibreChat) with support for `workspaceScopes` before enabling lanes on a worker. +##### Retiring stale linked worktrees + +Nothing else removes a task's worktree once its work is pushed, and each one +keeps its own dependencies and build output. A worker with lanes therefore +retires stale ones itself, with no configuration: a pass runs two minutes after +startup and every six hours after that, sooner while a backlog remains, and +also before managed environment setup would be deferred for low disk space. +Passes run in the background, never overlap, and never hold back requests. + +A worktree in a writable registered root is retired only when **all** of these +hold; otherwise it is kept and the reason is counted: + +- It verifies as a linked worktree of the checkout under `.worktrees/`, as for + lane admission. The checkout itself and worktrees elsewhere are never + touched. +- Neither the lane nor its checkout has a request in flight, and both this + worker's last use of the lane and the newest on-disk activity (the worktree + directory and its Git `HEAD`, `index`, `logs/HEAD`, `ORIG_HEAD` and + `FETCH_HEAD`) are older than the idle threshold, seven days by default. +- It has no modified tracked files and no untracked files the repository does + not ignore; no merge, rebase, cherry-pick, revert or bisect in progress; no + `git worktree lock`; and no quarantine on the lane or its checkout. +- Its `HEAD` commit, including a detached one, is contained in at least one + remote-tracking ref (`refs/remotes/*`), or its work is already in the + remote's default branch by content, as after a squash or rebase merge whose + remote branch was deleted. Content counts only when `HEAD` has no live + upstream (it is detached, never pushed, or its upstream ref is gone) and + either every commit beyond the default branch is patch-equivalent to one in + it (`git cherry` shows only `-`, with no merge commits), or the default + branch has identical content at every path the branch changed since their + merge base. The default branch is the remote's `HEAD`, else `main` or + `master`, as last fetched; the worker never fetches or calls a hosting API. + Commits beyond a live upstream are never treated as merged. + +Removal is `git worktree remove` **without** `--force`, so Git re-checks for +changes itself, followed by `git worktree prune`. Ignored files such as +`node_modules`, build output and ignored `.env` files go with the worktree; +that is the space being reclaimed. The branch is kept, so +`git worktree add .worktrees/ ` restores the worktree. A lane +request that arrives during removal waits for it and then fails as an unknown +worktree. Each pass inspects at most 128 idle worktrees and removes at most 32, +oldest first. Each pass logs one summary line, for example +`worktree retirement: retired 3, kept 12 (dirty 2, recent 8, unpushed 2), freed +about 4.1 GiB`; set `LIBRECHAT_CODE_LOG_LEVEL=debug` to log every kept worktree +and its reason. + +Pass `--no-worktree-retirement` or set `LIBRECHAT_CODE_WORKTREE_RETIREMENT=false` +to disable retirement. Change the idle threshold with `--worktree-idle-days ` +or `LIBRECHAT_CODE_WORKTREE_IDLE_DAYS=` (1 to 3650 days). + On an updated Code API, admission waits up to 30 seconds without the `X-LibreChat-Workspace-Queue-Wait-Ms` request header. A caller may advertise a positive integer millisecond allowance up to five minutes, capped by any server diff --git a/packages/code/src/cli.ts b/packages/code/src/cli.ts index cf05e263..6b450b64 100644 --- a/packages/code/src/cli.ts +++ b/packages/code/src/cli.ts @@ -43,6 +43,10 @@ import { NativeWorkspaceCommandPool } from './native-pool.js'; import { GitWorktreeWorkspaceTools, internalWorkspaceId } from './workspace-instances.js'; import { LINKED_WORKTREE_DIRECTORY, LinkedWorktreeWorkspaceTools } from './linked-worktrees.js'; import { GitWorktreeManager } from './worktrees.js'; +import { + WorktreeRetirementScheduler, + worktreeRetirementSettings, +} from './worktree-retirement.js'; import { captureWorkspaceRootIdentity } from './root-identity.js'; import { resolveNativeSrtCommandPolicy, @@ -812,6 +816,13 @@ async function run( if (linkedWorktreeLanes && process.platform === 'win32') { throw new Error('Linked worktree Git guard requires a POSIX host'); } + const worktreeRetirement = worktreeRetirementSettings({ + optOut: args.includes('--no-worktree-retirement'), + enabled: process.env.LIBRECHAT_CODE_WORKTREE_RETIREMENT, + idleDays: + option(args, '--worktree-idle-days') ?? + process.env.LIBRECHAT_CODE_WORKTREE_IDLE_DAYS, + }); if ( roots.length > 1 && process.env.LIBRECHAT_CODE_WORKSPACE_QUARANTINE_FILE?.trim() @@ -1301,6 +1312,41 @@ async function run( }); workspaceTools = linkedWorktreeTools; } + const retirementSources = roots.filter((root) => root.writable); + const debugLogs = + process.env.LIBRECHAT_CODE_LOG_LEVEL?.trim().toLowerCase() === 'debug'; + const worktreeRetirementScheduler = + linkedWorktreeTools && + worktreeRetirement.enabled && + retirementSources.length > 0 && + option(args, '--reset-workspace-quarantine') == null + ? new WorktreeRetirementScheduler({ + activity: linkedWorktreeTools, + idleMs: worktreeRetirement.idleMs, + sources: retirementSources.map((root) => ({ + workspaceId: root.id, + root: root.root, + identity: root.identity, + })), + async isQuarantined(selectedWorkspaceId, worktree) { + const source = roots.find((root) => root.id === selectedWorkspaceId); + if (!source) return true; + const path = + worktree == null + ? rootQuarantinePaths.get(selectedWorkspaceId)! + : defaultWorkspaceQuarantinePath({ + codeApiUrl, + workerId, + workspaceRoot: join(source.root, LINKED_WORKTREE_DIRECTORY, worktree), + }); + return (await loadWorkspaceMutationQuarantine(path)) != null; + }, + log(level, message) { + if (level === 'debug' && !debugLogs) return; + process.stdout.write(`librechat-code: ${message}\n`); + }, + }) + : undefined; if (workspaceTools && environments.length) { workspaceTools = new EnvironmentWorkspaceTools( workspaceTools, @@ -1375,6 +1421,9 @@ async function run( root: environment.definition.root, identity: roots.find(root => root.id === id)!.identity!, setup, receiptPath: preparationReceipt(environment.definition.root), + reclaimSpace: worktreeRetirementScheduler + ? () => worktreeRetirementScheduler.runNow() + : undefined, context: JSON.stringify([serializeNativeSrtCommandPolicy(commandPolicy), commandAllowedDomains, github.policyIdentity, nativeOptionsForWorkspace(id).resources]), signal: controller.signal, @@ -1404,10 +1453,12 @@ async function run( ); } } catch (error) { + await worktreeRetirementScheduler?.stop().catch(() => undefined); await nativeCommandSandbox?.close().catch(() => undefined); await fileRelaySupervisor?.stop().catch(() => undefined); throw error; } + worktreeRetirementScheduler?.start(); try { const worker = new BridgeWorker({ instructionDescriptors: () => localWorkspaceTools?.instructionDescriptors() ?? Promise.resolve(undefined), @@ -1617,6 +1668,7 @@ async function run( await worker.run(controller.signal); } finally { try { + await worktreeRetirementScheduler?.stop(); await nativeCommandSandbox?.close(); } finally { await fileRelaySupervisor?.stop(); diff --git a/packages/code/src/environment-preparation.ts b/packages/code/src/environment-preparation.ts index 8dbb8751..b17b4d32 100644 --- a/packages/code/src/environment-preparation.ts +++ b/packages/code/src/environment-preparation.ts @@ -37,6 +37,8 @@ export interface EnvironmentPreparationOptions { snapshotScope?: string; beforeMutation?(): Promise; storage?: EnvironmentStoragePolicy; + /** Frees reproducible storage when the storage floor would defer setup. */ + reclaimSpace?(): Promise; } /** Checkout-local reuse. Never transfers mutable installations between worktrees. */ @@ -87,7 +89,12 @@ async function prepareInLock( } if (options.snapshotStore && portable) { if (options.storage) - await assertPreparationSpace(options.root, options.storage); + await assertPreparationSpace( + options.root, + options.storage, + undefined, + options.reclaimSpace, + ); await options.beforeMutation?.(); if ( await restoreDependencySnapshot( @@ -118,7 +125,12 @@ async function prepareInLock( } } if (options.storage) - await assertPreparationSpace(options.root, options.storage); + await assertPreparationSpace( + options.root, + options.storage, + undefined, + options.reclaimSpace, + ); const result = await options.execute( options.setup.command, options.setup.timeoutMs, diff --git a/packages/code/src/linked-worktrees.ts b/packages/code/src/linked-worktrees.ts index e049aa6e..014b9787 100644 --- a/packages/code/src/linked-worktrees.ts +++ b/packages/code/src/linked-worktrees.ts @@ -32,6 +32,24 @@ const LINKED_WORKTREE_SHARED_GIT_PATHS = ['objects', 'refs', join('logs', 'refs' const LINKED_WORKTREE_LANE_LIMIT = 32; /** Git pointer files are a single line; anything larger is not one. */ const GIT_POINTER_MAX_BYTES = 4096; +/** Last-use timestamps kept per worker; the oldest are forgotten first. */ +const LINKED_WORKTREE_ACTIVITY_LIMIT = 4096; + +/** What this worker knows about lane use, for deciding when a worktree may be retired. */ +export interface LinkedWorktreeActivity { + /** When a request for the lane last started or finished in this process. */ + lastUsed(workspaceId: string, worktree: string): number | undefined; + /** + * Run `task` only while neither the lane nor its checkout has a request in + * flight. Lane requests that arrive meanwhile wait for it to finish, then + * verify the lane again; checkout requests are never held back. + */ + whileIdle( + workspaceId: string, + worktree: string, + task: () => Promise, + ): Promise<{ ran: true; value: T } | { ran: false }>; +} export interface LinkedWorktreeSource { root: string; @@ -206,7 +224,7 @@ function publicResult(result: WorkspaceToolResult, workspaceId: string): Workspa * checkout, so file tools and commands here run confined to that worktree while * sibling lanes run concurrently. Requests without a worktree pass through. */ -export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor { +export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, LinkedWorktreeActivity { readonly mutationFailuresAreAtomic?: true; readonly capabilities: WorkspaceToolExecutor['capabilities']; private readonly executors = new Map< @@ -216,6 +234,14 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor { private readonly commandRoots = new Map(); /** Verified lane roots by internal ID, least recently used first. */ private readonly lanes = new Map(); + /** Requests in flight by lane internal ID. */ + private readonly laneRequests = new Map(); + /** Requests in flight by checkout workspace ID, outside any lane. */ + private readonly checkoutRequests = new Map(); + /** Last request start or finish by lane internal ID, oldest first. */ + private readonly used = new Map(); + /** Lanes being retired; their requests wait for retirement to settle. */ + private readonly retiring = new Map>(); constructor(private readonly options: LinkedWorktreeWorkspaceToolsOptions) { this.mutationFailuresAreAtomic = options.delegate.mutationFailuresAreAtomic; @@ -337,29 +363,99 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor { return pool; } + lastUsed(workspaceId: string, worktree: string): number | undefined { + return this.used.get(linkedWorktreeWorkspaceId(workspaceId, worktree)); + } + + async whileIdle( + workspaceId: string, + worktree: string, + task: () => Promise, + ): Promise<{ ran: true; value: T } | { ran: false }> { + const internalId = linkedWorktreeWorkspaceId(workspaceId, worktree); + if ( + this.laneRequests.has(internalId) || + this.checkoutRequests.has(workspaceId) || + this.retiring.has(internalId) + ) { + return { ran: false }; + } + let settle!: () => void; + this.retiring.set(internalId, new Promise((resolve) => { settle = resolve; })); + try { + return { ran: true, value: await task() }; + } finally { + await this.release(internalId); + this.retiring.delete(internalId); + settle(); + } + } + + private touch(internalId: string): void { + this.used.delete(internalId); + this.used.set(internalId, Date.now()); + if (this.used.size <= LINKED_WORKTREE_ACTIVITY_LIMIT) return; + const oldest = this.used.keys().next().value; + if (oldest != null) this.used.delete(oldest); + } + + /** Count a request from its first synchronous step, so `whileIdle` never races its admission. */ + private async tracked( + requests: Map, + key: string, + task: () => Promise, + lane: boolean, + ): Promise { + requests.set(key, (requests.get(key) ?? 0) + 1); + if (lane) this.touch(key); + try { + return await task(); + } finally { + const remaining = (requests.get(key) ?? 1) - 1; + if (remaining > 0) requests.set(key, remaining); + else requests.delete(key); + if (lane) this.touch(key); + } + } + + private async inLane(workspaceId: string, worktree: string, task: () => Promise): Promise { + const internalId = linkedWorktreeWorkspaceId(workspaceId, worktree); + return await this.tracked(this.laneRequests, internalId, async () => { + await this.retiring.get(internalId); + return await task(); + }, true); + } + async execute(request: WorkspaceToolRequest, signal?: AbortSignal): Promise { if (request.worktree == null) { - return await this.options.delegate.execute(request, signal); + return await this.tracked( + this.checkoutRequests, + request.workspaceId, + () => this.options.delegate.execute(request, signal), + false, + ); } if (request.operation === 'execute_command' && request.environmentAction) { throw rejected('Environment action was not resolved by this worker'); } const { worktree, ...baseRequest } = request; - const { lane, source, internalId } = await this.resolveLane( - request.workspaceId, - worktree, - request.workspaceInstanceId, - ); - const laneRequest = { ...baseRequest, workspaceId: internalId } as WorkspaceToolRequest; - if (request.operation === 'execute_command') { - const pool = await this.registerCommandRoot(internalId, lane, source); - return publicResult( - await pool.execute(laneRequest as WorkspaceExecuteCommandRequest, signal), + return await this.inLane(request.workspaceId, worktree, async () => { + const { lane, source, internalId } = await this.resolveLane( request.workspaceId, + worktree, + request.workspaceInstanceId, ); - } - const executor = await this.fileExecutor(internalId, lane, source); - return publicResult(await executor.execute(laneRequest, signal), request.workspaceId); + const laneRequest = { ...baseRequest, workspaceId: internalId } as WorkspaceToolRequest; + if (request.operation === 'execute_command') { + const pool = await this.registerCommandRoot(internalId, lane, source); + return publicResult( + await pool.execute(laneRequest as WorkspaceExecuteCommandRequest, signal), + request.workspaceId, + ); + } + const executor = await this.fileExecutor(internalId, lane, source); + return publicResult(await executor.execute(laneRequest, signal), request.workspaceId); + }); } async executeProgrammatic( @@ -369,18 +465,26 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor { ): Promise { const worktree = request.body.workspace_worktree; if (worktree == null) { - if (!this.options.programmaticDelegate) { + const delegate = this.options.programmaticDelegate; + if (!delegate) { throw new WorkspaceToolError('Workspace programmatic execution is unavailable', 'COMMAND_DISABLED'); } - return await this.options.programmaticDelegate.executeProgrammatic(workspaceId, request, signal); + return await this.tracked( + this.checkoutRequests, + workspaceId, + () => delegate.executeProgrammatic(workspaceId, request, signal), + false, + ); } - const { lane, source, internalId } = await this.resolveLane( - workspaceId, - worktree, - request.body.workspace_instance_id, - ); - const pool = await this.registerCommandRoot(internalId, lane, source); - const { workspace_worktree: _worktree, ...body } = request.body; - return await pool.executeProgrammatic(internalId, { ...request, body }, signal); + return await this.inLane(workspaceId, worktree, async () => { + const { lane, source, internalId } = await this.resolveLane( + workspaceId, + worktree, + request.body.workspace_instance_id, + ); + const pool = await this.registerCommandRoot(internalId, lane, source); + const { workspace_worktree: _worktree, ...body } = request.body; + return await pool.executeProgrammatic(internalId, { ...request, body }, signal); + }); } } diff --git a/packages/code/src/snapshot-lifecycle.test.ts b/packages/code/src/snapshot-lifecycle.test.ts index 33e37c2d..8074fd11 100644 --- a/packages/code/src/snapshot-lifecycle.test.ts +++ b/packages/code/src/snapshot-lifecycle.test.ts @@ -116,6 +116,54 @@ test('storage policy is opt-in, bounded and fails before a setup command', async assert.equal(executions, 0); }); +test('low space reclaims once before deferring setup', async t => { + const policy = { minFreeBytes: 5, setupReserveBytes: 2 }; + let free = 6n; + let reclaims = 0; + await assertPreparationSpace('/', policy, async () => free, async () => { + reclaims++; + free = 7n; + }); + assert.equal(reclaims, 1); + await assertPreparationSpace('/', policy, async () => free, async () => { + reclaims++; + }); + assert.equal(reclaims, 1, 'enough space never triggers reclamation'); + await assert.rejects( + assertPreparationSpace('/', policy, async () => 6n, async () => { + reclaims++; + }), + /no setup command was started/, + ); + assert.equal(reclaims, 2); + + const { root } = await fixture(t); + const checkout = join(root, 'checkout'); + await mkdir(checkout); + let executions = 0; + let reclaimed = 0; + await assert.rejects( + prepareCodeEnvironment({ + root: checkout, + identity: await captureWorkspaceRootIdentity(checkout), + setup: { command: 'install', timeoutMs: 1000 }, + receiptPath: join(root, 'receipt'), + context: '', + storage: { minFreeBytes: Number.MAX_SAFE_INTEGER, setupReserveBytes: 0 }, + reclaimSpace: async () => { + reclaimed++; + }, + execute: async () => { + executions++; + return { exitCode: 0, timedOut: false }; + }, + }), + /deferred/, + ); + assert.equal(reclaimed, 1); + assert.equal(executions, 0); +}); + test('dry-run is non-destructive; cleanup removes only expired owned entries and retains unknown or linked data', async t => { const { store, add, a, b, c, root } = await fixture(t); const old = await add(a, 4, 10_000); diff --git a/packages/code/src/snapshot-lifecycle.ts b/packages/code/src/snapshot-lifecycle.ts index dfa5926e..66ed39dd 100644 --- a/packages/code/src/snapshot-lifecycle.ts +++ b/packages/code/src/snapshot-lifecycle.ts @@ -63,7 +63,11 @@ export function parseEnvironmentStorage( throw new Error('Invalid environment storage policy'); return policy as EnvironmentStoragePolicy; } -/** Soft admission for managed preparation, not a reservation or arbitrary-write quota. */ +/** + * Soft admission for managed preparation, not a reservation or arbitrary-write + * quota. Below the floor, `reclaim` (when supplied) may free reproducible + * storage once before the space is measured again. + */ export async function assertPreparationSpace( root: string, policy: EnvironmentStoragePolicy, @@ -71,14 +75,18 @@ export async function assertPreparationSpace( const status = await statfs(path, { bigint: true }); return status.bavail * status.bsize; }, + reclaim?: () => Promise, ): Promise { - if ( - (await available(root)) < - BigInt(policy.minFreeBytes) + BigInt(policy.setupReserveBytes) - ) - throw new Error( - 'Managed environment preparation deferred: free disk is below the configured floor plus setup reserve. Clean reproducible artifacts or expand storage; no setup command was started.', - ); + const floor = + BigInt(policy.minFreeBytes) + BigInt(policy.setupReserveBytes); + if ((await available(root)) >= floor) return; + if (reclaim) { + await reclaim(); + if ((await available(root)) >= floor) return; + } + throw new Error( + 'Managed environment preparation deferred: free disk is below the configured floor plus setup reserve. Clean reproducible artifacts or expand storage; no setup command was started.', + ); } export const SNAPSHOT_MANIFEST = '.snapshot.json'; diff --git a/packages/code/src/worktree-retirement.test.ts b/packages/code/src/worktree-retirement.test.ts new file mode 100644 index 00000000..427e7e20 --- /dev/null +++ b/packages/code/src/worktree-retirement.test.ts @@ -0,0 +1,520 @@ +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { mkdir, mkdtemp, readFile, realpath, rm, stat, utimes, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; +import test from 'node:test'; + +import { LinkedWorktreeWorkspaceTools } from './linked-worktrees.js'; +import { LocalWorkspaceTools, WorkspaceToolError } from './workspace.js'; +import { + WorktreeRetirementScheduler, + describeWorktreeRetirement, + retireStaleWorktrees, + worktreeRetirementSettings, +} from './worktree-retirement.js'; + +import type { NativeWorkspaceCommandPool } from './native-pool.js'; +import type { NativeProcessSandboxOptions } from './native-process.js'; +import type { WorktreeRetirementOptions } from './worktree-retirement.js'; + +const execFileAsync = promisify(execFile); +const DAY_MS = 24 * 60 * 60 * 1000; + +async function git(cwd: string, ...args: string[]): Promise { + const { stdout } = await execFileAsync( + 'git', + ['-c', 'user.name=test', '-c', 'user.email=test@example.com', '-c', 'commit.gpgsign=false', ...args], + { cwd }, + ); + return stdout.trim(); +} + +async function exists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +/** A checkout on `main` with an `origin` bare remote and an ignored `node_modules`. */ +async function repository(t: test.TestContext): Promise { + const parent = await realpath(await mkdtemp(join(tmpdir(), 'worktree-retirement-'))); + t.after(() => rm(parent, { recursive: true, force: true })); + const remote = join(parent, 'remote.git'); + await git(parent, 'init', '-q', '--bare', remote); + const root = join(parent, 'repo'); + await mkdir(root); + await git(root, 'init', '-q', '-b', 'main'); + await writeFile(join(root, 'README.md'), 'root\n'); + await writeFile(join(root, '.gitignore'), 'node_modules/\n.worktrees/\n'); + await git(root, 'add', '.'); + await git(root, 'commit', '-q', '-m', 'init'); + await git(root, 'remote', 'add', 'origin', remote); + await git(root, 'push', '-q', 'origin', 'main'); + await mkdir(join(root, '.worktrees')); + return root; +} + +/** A task worktree with one commit of its own and ignored dependencies. */ +async function worktree(root: string, name: string, push = true): Promise { + await git(root, 'worktree', 'add', '-q', '-b', name, `.worktrees/${name}`); + const path = join(root, '.worktrees', name); + await writeFile(join(path, `${name}.txt`), `${name}\n`); + await git(path, 'add', '.'); + await git(path, 'commit', '-q', '-m', name); + if (push) await git(path, 'push', '-q', 'origin', name); + await mkdir(join(path, 'node_modules', 'dependency'), { recursive: true }); + await writeFile(join(path, 'node_modules', 'dependency', 'index.js'), 'module.exports = 1;\n'); + return path; +} + +/** Backdate every on-disk activity signal, as if nobody had touched the worktree for `days`. */ +async function age(root: string, name: string, days = 30): Promise { + const when = new Date(Date.now() - days * DAY_MS); + const metadata = join(root, '.git', 'worktrees', name); + const paths = [ + join(root, '.worktrees', name), + metadata, + ...['HEAD', 'index', join('logs', 'HEAD'), 'ORIG_HEAD', 'FETCH_HEAD'].map((path) => join(metadata, path)), + ]; + for (const path of paths) { + await utimes(path, when, when).catch(() => undefined); + } +} + +function sources(root: string): WorktreeRetirementOptions['sources'] { + return [{ workspaceId: 'repo', root }]; +} + +/** A command pool whose lane commands wait until released. */ +function blockingPool(): { + pool: NativeWorkspaceCommandPool; + started: Promise; + finish: () => void; +} { + let started!: () => void; + let finish!: () => void; + const startedPromise = new Promise((resolve) => { started = resolve; }); + const finished = new Promise((resolve) => { finish = resolve; }); + const pool = { + async registerRoot() {}, + async unregisterRoot() {}, + async execute(request: { workspaceId: string }) { + started(); + await finished; + return { + protocolVersion: 1, + operation: 'execute_command', + workspaceId: request.workspaceId, + exitCode: 0, + stdout: '', + stderr: '', + truncated: false, + timedOut: false, + }; + }, + } as unknown as NativeWorkspaceCommandPool; + return { pool, started: startedPromise, finish }; +} + +async function laneTools(root: string, pool?: NativeWorkspaceCommandPool): Promise { + const delegate = await LocalWorkspaceTools.create({ + repositoryInstructions: false, + workspaces: [{ id: 'repo', root, writable: true }], + }); + return new LinkedWorktreeWorkspaceTools({ + commandPool: pool, + delegate, + sources: new Map([ + ['repo', { + root, + command: { workspaceRoot: root } as NativeProcessSandboxOptions, + repositoryInstructions: false, + writable: true, + }], + ]), + }); +} + +const laneCommand = (worktree: string) => ({ + protocolVersion: 1 as const, + operation: 'execute_command' as const, + workspaceId: 'repo', + worktree, + command: 'true', +}); + +const pause = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +test('retires a clean, pushed, idle worktree and keeps its branch for recovery', async (t) => { + const root = await repository(t); + const path = await worktree(root, 'done'); + await age(root, 'done'); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, ['repo:done']); + assert.deepEqual(summary.kept, {}); + assert.equal(await exists(path), false, 'the worktree and its ignored files are gone'); + assert.equal(await exists(join(root, '.git', 'worktrees', 'done')), false); + assert.doesNotMatch(await git(root, 'worktree', 'list', '--porcelain'), /\.worktrees\/done/); + const branch = await git(root, 'rev-parse', '--verify', 'refs/heads/done'); + assert.equal(branch, await git(root, 'rev-parse', '--verify', 'refs/remotes/origin/done')); + + await git(root, 'worktree', 'add', '-q', '.worktrees/done', 'done'); + assert.equal(await readFile(join(path, 'done.txt'), 'utf8'), 'done\n'); + assert.equal(await git(root, 'status', '--porcelain'), '', 'the main checkout is untouched'); + assert.equal(await git(root, 'branch', '--show-current'), 'main'); +}); + +test('keeps every worktree that could lose work or is not a linked worktree', async (t) => { + const root = await repository(t); + const worktrees = join(root, '.worktrees'); + + const dirty = await worktree(root, 'dirty'); + await writeFile(join(dirty, 'dirty.txt'), 'changed\n'); + const untracked = await worktree(root, 'untracked'); + await writeFile(join(untracked, 'notes.txt'), 'not yet added\n'); + await worktree(root, 'unpushed', false); + await worktree(root, 'locked'); + await git(root, 'worktree', 'lock', '.worktrees/locked'); + const merging = await worktree(root, 'merging'); + await worktree(root, 'side'); + await git(merging, 'merge', '-q', '--no-commit', '--no-ff', 'side'); + await worktree(root, 'recent'); + const detached = await worktree(root, 'detached'); + await git(detached, 'checkout', '-q', '--detach'); + await writeFile(join(detached, 'local.txt'), 'local\n'); + await git(detached, 'add', '.'); + await git(detached, 'commit', '-q', '-m', 'detached and unpushed'); + await mkdir(join(worktrees, 'forged')); + await writeFile(join(worktrees, 'forged', '.git'), `gitdir: ${join(root, '.git', 'worktrees', 'side')}\n`); + await mkdir(join(worktrees, 'plain')); + for (const name of ['dirty', 'untracked', 'unpushed', 'locked', 'merging', 'side', 'detached']) { + await age(root, name); + } + await age(root, 'forged'); + await age(root, 'plain'); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, ['repo:side']); + assert.deepEqual(summary.kept, { + dirty: 2, + unpushed: 2, + locked: 1, + operation: 1, + recent: 1, + unverified: 2, + }); + for (const name of ['dirty', 'untracked', 'unpushed', 'locked', 'merging', 'recent', 'detached', 'forged', 'plain']) { + assert.ok(await exists(join(worktrees, name)), `${name} is kept`); + } + assert.equal(await readFile(join(dirty, 'dirty.txt'), 'utf8'), 'changed\n'); + assert.equal(await readFile(join(untracked, 'notes.txt'), 'utf8'), 'not yet added\n'); + assert.ok(await exists(join(root, '.git', 'worktrees', 'merging', 'MERGE_HEAD'))); + assert.equal(await git(root, 'status', '--porcelain'), ''); + assert.equal(await readFile(join(root, 'README.md'), 'utf8'), 'root\n'); +}); + +test('a detached HEAD is retired only when a remote-tracking ref contains it', async (t) => { + const root = await repository(t); + const path = await worktree(root, 'review'); + await git(path, 'checkout', '-q', '--detach'); + await age(root, 'review'); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, ['repo:review']); + assert.equal(await exists(path), false); +}); + +test('a squash- or rebase-merged branch whose remote branch was deleted is retired; unmerged work is kept', async (t) => { + const root = await repository(t); + const parent = join(root, '..'); + const tracked = async (name: string): Promise => { + const path = await worktree(root, name); + await git(path, 'branch', '-q', '--set-upstream-to', `origin/${name}`); + return path; + }; + const squashed = await tracked('squashed'); + await writeFile(join(squashed, 'second.txt'), 'second\n'); + await git(squashed, 'add', '.'); + await git(squashed, 'commit', '-q', '-m', 'second'); + await git(squashed, 'push', '-q', 'origin', 'squashed'); + await tracked('rebased'); + await tracked('abandoned'); + const review = 'review-16677-1f35df8'; + await git(root, 'worktree', 'add', '-q', '--detach', `.worktrees/${review}`, await git(squashed, 'rev-parse', 'HEAD')); + await git(root, 'worktree', 'add', '-q', '-b', 'ahead', '.worktrees/ahead'); + const ahead = join(root, '.worktrees', 'ahead'); + await git(ahead, 'push', '-q', '-u', 'origin', 'ahead'); + await writeFile(join(ahead, 'later.txt'), 'later\n'); + await git(ahead, 'add', '.'); + await git(ahead, 'commit', '-q', '-m', 'later, also landed on main but never pushed here'); + + const hosting = join(parent, 'hosting'); + await git(parent, 'clone', '-q', join(parent, 'remote.git'), hosting); + await git(hosting, 'merge', '-q', '--squash', 'origin/squashed'); + await git(hosting, 'commit', '-q', '-m', 'squash merge'); + await git(hosting, 'cherry-pick', 'origin/rebased'); + await writeFile(join(hosting, 'later.txt'), 'later\n'); + await git(hosting, 'add', '.'); + await git(hosting, 'commit', '-q', '-m', 'later'); + await git(hosting, 'push', '-q', 'origin', 'main'); + await git(hosting, 'push', '-q', 'origin', '--delete', 'squashed', 'rebased', 'abandoned'); + await git(root, 'fetch', '-q', '--prune', 'origin'); + for (const name of ['squashed', 'rebased', 'abandoned', review, 'ahead']) await age(root, name); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual([...summary.retired].sort(), ['repo:rebased', `repo:${review}`, 'repo:squashed']); + assert.deepEqual(summary.kept, { unpushed: 2 }); + assert.ok(await exists(join(root, '.worktrees', 'abandoned')), 'unmerged work with a deleted remote is kept'); + assert.ok(await exists(ahead), 'commits beyond a live upstream are kept even when main has the same change'); + for (const branch of ['squashed', 'rebased', 'abandoned', 'ahead']) { + await git(root, 'rev-parse', '--verify', `refs/heads/${branch}`); + } +}); + +test('never touches the main checkout or worktrees outside .worktrees', async (t) => { + const root = await repository(t); + const outside = join(root, '..', 'outside'); + await git(root, 'worktree', 'add', '-q', '-b', 'outside', outside); + await git(outside, 'push', '-q', 'origin', 'outside'); + const nested = join(root, 'nested'); + await git(root, 'worktree', 'add', '-q', '-b', 'nested', nested); + await git(nested, 'push', '-q', 'origin', 'nested'); + const old = new Date(Date.now() - 30 * DAY_MS); + for (const path of [root, join(root, '.git'), join(root, '.git', 'HEAD'), join(root, '.git', 'index'), outside, nested]) { + await utimes(path, old, old); + } + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, []); + assert.ok(await exists(join(root, 'README.md'))); + assert.ok(await exists(outside)); + assert.ok(await exists(nested)); + assert.equal((await git(root, 'worktree', 'list', '--porcelain')).match(/^worktree /gm)?.length, 3); +}); + +test('this worker\'s own lane use counts as activity, and an active lane is never retired', async (t) => { + const root = await repository(t); + const path = await worktree(root, 'busy'); + await age(root, 'busy'); + const { pool, started, finish } = blockingPool(); + const tools = await laneTools(root, pool); + + const running = tools.execute(laneCommand('busy')); + await started; + await pause(5); + const during = await retireStaleWorktrees({ sources: sources(root), activity: tools, idleMs: 1 }); + assert.deepEqual(during.kept, { active: 1 }); + assert.ok(await exists(path)); + + finish(); + await running; + const recent = await retireStaleWorktrees({ sources: sources(root), activity: tools }); + assert.deepEqual(recent.kept, { recent: 1 }, 'the default threshold counts this process\'s last use'); + assert.ok(await exists(path)); + + await pause(5); + const after = await retireStaleWorktrees({ sources: sources(root), activity: tools, idleMs: 1 }); + assert.deepEqual(after.retired, ['repo:busy']); + assert.equal(await exists(path), false); + await assert.rejects( + tools.execute(laneCommand('busy')), + (error: unknown) => error instanceof WorkspaceToolError && error.code === 'INVALID_REQUEST', + ); +}); + +test('a lane request that arrives during retirement waits, then finds the lane gone', async (t) => { + const root = await repository(t); + await worktree(root, 'leaving'); + let executions = 0; + const pool = { + async registerRoot() {}, + async unregisterRoot() {}, + async execute() { + executions += 1; + throw new Error('a retired lane must not run commands'); + }, + } as unknown as NativeWorkspaceCommandPool; + const tools = await laneTools(root, pool); + let release!: () => void; + const gate = new Promise((resolve) => { release = resolve; }); + + const retiring = tools.whileIdle('repo', 'leaving', async () => { + await gate; + await git(root, 'worktree', 'remove', '.worktrees/leaving'); + return 'removed'; + }); + let settled = false; + const request = tools.execute(laneCommand('leaving')); + request.catch(() => undefined).finally(() => { settled = true; }); + await pause(20); + assert.equal(settled, false, 'the lane request waits for retirement'); + assert.deepEqual(await tools.whileIdle('repo', 'leaving', async () => 'again'), { ran: false }); + + release(); + assert.deepEqual(await retiring, { ran: true, value: 'removed' }); + await assert.rejects( + request, + (error: unknown) => error instanceof WorkspaceToolError && error.code === 'INVALID_REQUEST', + ); + assert.equal(executions, 0); +}); + +test('a checkout request in flight defers retirement of its lanes', async (t) => { + const root = await repository(t); + const path = await worktree(root, 'quiet'); + await age(root, 'quiet'); + let release!: () => void; + let entered!: () => void; + const inside = new Promise((resolve) => { entered = resolve; }); + const gate = new Promise((resolve) => { release = resolve; }); + const delegate = await LocalWorkspaceTools.create({ + repositoryInstructions: false, + workspaces: [{ id: 'repo', root, writable: true }], + }); + const tools = new LinkedWorktreeWorkspaceTools({ + delegate: { + capabilities: delegate.capabilities, + async execute(request, signal) { + entered(); + await gate; + return await delegate.execute(request, signal); + }, + }, + sources: new Map([['repo', { root, repositoryInstructions: false, writable: true }]]), + }); + + const reading = tools.execute({ protocolVersion: 1, operation: 'read_file', workspaceId: 'repo', path: 'README.md' }); + await inside; + const during = await retireStaleWorktrees({ sources: sources(root), activity: tools }); + assert.deepEqual(during.kept, { active: 1 }); + release(); + await reading; + assert.ok(await exists(path)); + + const after = await retireStaleWorktrees({ sources: sources(root), activity: tools }); + assert.deepEqual(after.retired, ['repo:quiet']); +}); + +test('quarantined lanes and checkouts are left for the operator', async (t) => { + const root = await repository(t); + await worktree(root, 'held'); + await worktree(root, 'free'); + await age(root, 'held'); + await age(root, 'free'); + + const checkoutHeld = await retireStaleWorktrees({ + sources: sources(root), + isQuarantined: async (_workspaceId, worktree) => worktree == null, + }); + assert.deepEqual(checkoutHeld, { retired: [], kept: {}, freedBytes: 0, truncated: false }); + + const laneHeld = await retireStaleWorktrees({ + sources: sources(root), + isQuarantined: async (_workspaceId, worktree) => worktree === 'held', + }); + assert.deepEqual(laneHeld.retired, ['repo:free']); + assert.deepEqual(laneHeld.kept, { quarantined: 1 }); + assert.ok(await exists(join(root, '.worktrees', 'held'))); +}); + +test('one worktree failing does not stop the pass', async (t) => { + const root = await repository(t); + await worktree(root, 'broken'); + await worktree(root, 'fine'); + await rm(join(root, '.git', 'worktrees', 'broken', 'HEAD')); + await age(root, 'broken'); + await age(root, 'fine'); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, ['repo:fine']); + assert.equal(Object.values(summary.kept).reduce((total, count) => total + (count ?? 0), 0), 1); + assert.ok(await exists(join(root, '.worktrees', 'broken'))); +}); + +test('worktrees kept for lasting reasons cannot starve the rest of a backlog', async (t) => { + const root = await repository(t); + for (const [index, name] of ['first', 'second', 'third'].entries()) { + await worktree(root, name, false); + await age(root, name, 40 - index); + } + await worktree(root, 'newest'); + await age(root, 'newest', 10); + const rotation = new Set(); + const options = { sources: sources(root), rotation, limits: { inspect: 2 } }; + + const first = await retireStaleWorktrees(options); + assert.deepEqual(first.retired, []); + assert.deepEqual(first.kept, { unpushed: 2, deferred: 2 }); + assert.equal(first.truncated, true, 'uninspected worktrees remain in this rotation'); + + const second = await retireStaleWorktrees(options); + assert.deepEqual(second.retired, ['repo:newest']); + assert.deepEqual(second.kept, { unpushed: 1, deferred: 2 }); + assert.equal(second.truncated, false, 'the rotation is complete'); + + const third = await retireStaleWorktrees(options); + assert.deepEqual(third.kept, { unpushed: 2, deferred: 1 }, 'a new rotation starts from the oldest'); + assert.equal(third.truncated, true); +}); + +test('retirement is on by default and can be disabled or retuned', () => { + assert.deepEqual(worktreeRetirementSettings({ optOut: false }), { enabled: true, idleMs: 7 * DAY_MS }); + assert.equal(worktreeRetirementSettings({ optOut: true }).enabled, false); + assert.equal(worktreeRetirementSettings({ optOut: false, enabled: 'FALSE' }).enabled, false); + assert.equal(worktreeRetirementSettings({ optOut: true, enabled: 'true' }).enabled, false); + assert.equal(worktreeRetirementSettings({ optOut: false, enabled: ' ' }).enabled, true); + assert.equal(worktreeRetirementSettings({ optOut: false, idleDays: '14' }).idleMs, 14 * DAY_MS); + assert.throws(() => worktreeRetirementSettings({ optOut: false, enabled: 'off' }), /must be true or false/); + for (const idleDays of ['0', '-1', '1.5', 'week', '3651']) { + assert.throws(() => worktreeRetirementSettings({ optOut: false, idleDays }), /IDLE_DAYS/); + } +}); + +test('scheduled passes never overlap and stop cleanly', async (t) => { + const root = await repository(t); + await worktree(root, 'scheduled'); + await age(root, 'scheduled'); + const messages: string[] = []; + const scheduler = new WorktreeRetirementScheduler({ + sources: sources(root), + startDelayMs: 60_000, + log: (level, message) => { + if (level === 'info') messages.push(message); + }, + }); + scheduler.start(); + + const first = scheduler.runNow(); + assert.equal(scheduler.runNow(), first, 'a pass requested while one runs joins it'); + const summary = await first; + assert.deepEqual(summary?.retired, ['repo:scheduled']); + assert.equal(messages.length, 1); + assert.match(messages[0]!, /^worktree retirement: retired 1, kept 0(, freed about .+)?$/); + + await scheduler.stop(); + assert.equal(await scheduler.runNow(), undefined, 'a stopped scheduler runs nothing'); +}); + +test('summaries count reasons and note a remaining backlog', () => { + assert.equal( + describeWorktreeRetirement({ + retired: ['repo:a', 'repo:b'], + kept: { unpushed: 2, dirty: 1, recent: 4 }, + freedBytes: 3 * 1024 ** 3, + truncated: true, + }), + 'worktree retirement: retired 2, kept 7 (dirty 1, recent 4, unpushed 2), freed about 3.0 GiB, more next pass', + ); +}); diff --git a/packages/code/src/worktree-retirement.ts b/packages/code/src/worktree-retirement.ts new file mode 100644 index 00000000..d13cb3d8 --- /dev/null +++ b/packages/code/src/worktree-retirement.ts @@ -0,0 +1,679 @@ +import { execFile } from 'node:child_process'; +import { lstat, opendir, statfs } from 'node:fs/promises'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; + +import { LINKED_WORKTREE_DIRECTORY, verifyLinkedWorktree } from './linked-worktrees.js'; +import { isValidLinkedWorktreeName } from './protocol.js'; + +import type { LinkedWorktreeActivity, VerifiedLinkedWorktree } from './linked-worktrees.js'; +import type { WorkspaceRootIdentity } from './root-identity.js'; + +const execFileAsync = promisify(execFile); + +const DAY_MS = 24 * 60 * 60 * 1000; +export const DEFAULT_WORKTREE_IDLE_DAYS = 7; +const MAX_WORKTREE_IDLE_DAYS = 3650; +export const WORKTREE_RETIREMENT_INTERVAL_MS = 6 * 60 * 60 * 1000; +const WORKTREE_RETIREMENT_START_DELAY_MS = 2 * 60 * 1000; +/** A pass that hit a bound continues soon instead of waiting a full interval. */ +const WORKTREE_RETIREMENT_CONTINUATION_MS = 5 * 60 * 1000; +/** Directory entries read beneath one checkout's `.worktrees` per pass. */ +const SCAN_LIMIT = 4096; +/** Idle worktrees whose Git state is inspected per pass, oldest first. */ +const INSPECT_LIMIT = 128; +/** Worktrees removed per pass. */ +const RETIRE_LIMIT = 32; +const GIT_TIMEOUT_MS = 30_000; +/** + * Removal deletes ignored build output too, which can take a while. It is never + * cut short: a half-deleted worktree loses its `.git` file and can no longer be + * verified, retired or recognized. + */ +const GIT_REMOVE_TIMEOUT_MS = 30 * 60_000; +const GIT_OUTPUT_LIMIT = 64 * 1024; +/** Path and commit lists between a stale branch and its default branch can be long. */ +const GIT_LIST_LIMIT = 16 * 1024 * 1024; +/** Per-worktree Git state that marks a merge, rebase, cherry-pick, revert or bisect in progress. */ +const OPERATION_STATE = [ + 'MERGE_HEAD', + 'CHERRY_PICK_HEAD', + 'REVERT_HEAD', + 'BISECT_LOG', + 'BISECT_START', + 'rebase-merge', + 'rebase-apply', + 'sequencer', +]; +/** Per-worktree Git metadata that Git rewrites whenever the worktree is used. */ +const ACTIVITY_STATE = ['HEAD', 'index', join('logs', 'HEAD'), 'ORIG_HEAD', 'FETCH_HEAD']; + +export type WorktreeKeptReason = + | 'unverified' + | 'recent' + | 'quarantined' + | 'locked' + | 'operation' + | 'no-commit' + | 'dirty' + | 'unpushed' + | 'active' + | 'changed' + | 'failed' + | 'deferred'; + +export interface WorktreeRetirementSource { + workspaceId: string; + root: string; + identity?: WorkspaceRootIdentity; +} + +export interface WorktreeRetirementOptions { + sources: readonly WorktreeRetirementSource[]; + /** Lane use in this worker; without it, only on-disk activity counts. */ + activity?: LinkedWorktreeActivity; + idleMs?: number; + /** A quarantined checkout or lane awaits an operator and is never retired. */ + isQuarantined?: (workspaceId: string, worktree?: string) => Promise; + log?: (level: 'debug' | 'info', message: string) => void; + /** + * Idle worktrees already inspected in the current rotation. A pass inspects + * the others first and records what it inspects, so worktrees kept for a + * lasting reason cannot starve the rest of a backlog larger than one pass. + */ + rotation?: Set; + /** Per-pass bounds; the defaults suit production. */ + limits?: { inspect?: number; retire?: number }; +} + +export interface WorktreeRetirementSummary { + /** `:` for each retired worktree. */ + retired: string[]; + kept: Partial>; + /** Free-space gain on the checkouts' filesystems across the pass; other writers make it approximate. */ + freedBytes: number; + /** Idle worktrees remain that this rotation has not inspected yet. */ + truncated: boolean; +} + +export interface WorktreeRetirementSettings { + enabled: boolean; + idleMs: number; +} + +interface Candidate { + source: WorktreeRetirementSource; + name: string; + lane: VerifiedLinkedWorktree; + metadata: string; + /** Newest on-disk activity, re-read under the reservation to detect use during inspection. */ + diskActiveAt: number; + /** This worker's own last use of the lane, re-read under the reservation. */ + usedAt?: number; + /** Newest of on-disk activity and this worker's own lane use. */ + activeAt: number; +} + +type Kept = { kept: WorktreeKeptReason; detail?: string }; +type Verdict = Kept | { head: string; basis: string }; + +/** + * Resolve the operator's choice. Retirement is on unless explicitly disabled; + * malformed values fail startup rather than silently choosing either way. + */ +export function worktreeRetirementSettings(input: { + optOut: boolean; + enabled?: string; + idleDays?: string; +}): WorktreeRetirementSettings { + const enabled = input.enabled?.trim().toLowerCase() ?? ''; + if (enabled !== '' && enabled !== 'true' && enabled !== 'false') { + throw new Error('LIBRECHAT_CODE_WORKTREE_RETIREMENT must be true or false'); + } + const idleDays = input.idleDays?.trim() ?? ''; + const days = idleDays === '' ? DEFAULT_WORKTREE_IDLE_DAYS : Number(idleDays); + if (!Number.isSafeInteger(days) || days < 1 || days > MAX_WORKTREE_IDLE_DAYS) { + throw new Error( + `LIBRECHAT_CODE_WORKTREE_IDLE_DAYS must be an integer between 1 and ${MAX_WORKTREE_IDLE_DAYS}`, + ); + } + return { enabled: !input.optOut && enabled !== 'false', idleMs: days * DAY_MS }; +} + +function gitEnvironment(): NodeJS.ProcessEnv { + return { + PATH: process.env.PATH, + SYSTEMROOT: process.env.SYSTEMROOT, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_TERMINAL_PROMPT: '0', + GIT_OPTIONAL_LOCKS: '0', + LC_ALL: 'C', + }; +} + +async function git( + cwd: string, + args: string[], + signal?: AbortSignal, + timeout = GIT_TIMEOUT_MS, + maxBuffer = GIT_OUTPUT_LIMIT, +): Promise { + const { stdout } = await execFileAsync( + 'git', + ['--no-optional-locks', '-C', cwd, '-c', 'core.fsmonitor=false', ...args], + { encoding: 'utf8', env: gitEnvironment(), maxBuffer, signal, timeout }, + ); + return stdout; +} + +/** The commit a ref names, or undefined when it does not resolve. */ +async function resolveCommit(cwd: string, ref: string, signal?: AbortSignal): Promise { + try { + const commit = (await git(cwd, ['rev-parse', '--verify', '--quiet', `${ref}^{commit}`], signal)).trim(); + return /^[0-9a-f]{40,64}$/.test(commit) ? commit : undefined; + } catch { + signal?.throwIfAborted(); + return undefined; + } +} + +function pathList(output: string): string[] { + return output.split('\0').filter(Boolean); +} + +/** The remote's default branch as last fetched: its `HEAD` symref, else `main` or `master`. */ +async function remoteDefaultBranch( + checkout: string, + remote: string, + signal?: AbortSignal, +): Promise<{ ref: string; commit: string } | undefined> { + let symbolic: string | undefined; + try { + symbolic = (await git(checkout, ['symbolic-ref', '-q', `refs/remotes/${remote}/HEAD`], signal)).trim(); + } catch { + signal?.throwIfAborted(); + } + const refs = [ + ...(symbolic?.startsWith(`refs/remotes/${remote}/`) ? [symbolic] : []), + `refs/remotes/${remote}/main`, + `refs/remotes/${remote}/master`, + ]; + for (const ref of refs) { + const commit = await resolveCommit(checkout, ref, signal); + if (commit) return { ref: ref.replace(/^refs\/remotes\//, ''), commit }; + } + return undefined; +} + +/** + * Whether a HEAD that no remote-tracking ref contains is nonetheless already + * in its remote's default branch by content, as after a squash or rebase merge + * whose remote branch was then deleted. Only a HEAD without a live upstream + * qualifies: detached, never pushed, or whose upstream ref is gone. A live + * upstream that lacks HEAD means unpushed commits. Content counts when every + * commit beyond the default branch is patch-equivalent to one in it (and none + * is a merge, which could carry its own changes), or when the default branch + * has identical content at every path the branch changed since their merge + * base. Local refs only; nothing is fetched. + */ +async function mergedByContent( + lane: VerifiedLinkedWorktree, + head: string, + signal?: AbortSignal, +): Promise { + const checkout = lane.checkoutRoot; + let branch: string | undefined; + try { + branch = (await git(lane.root, ['symbolic-ref', '-q', 'HEAD'], signal)).trim() || undefined; + } catch { + signal?.throwIfAborted(); + } + let remote = 'origin'; + if (branch?.startsWith('refs/heads/')) { + const [upstream = '', upstreamRemote = ''] = ( + await git(checkout, ['for-each-ref', '--format=%(upstream)%00%(upstream:remotename)', branch], signal) + ) + .trim() + .split('\0'); + if (upstream) { + if (await resolveCommit(checkout, upstream, signal)) return undefined; + if (/^[A-Za-z0-9._-]+$/.test(upstreamRemote)) remote = upstreamRemote; + } + } + const target = await remoteDefaultBranch(checkout, remote, signal); + if (!target) return undefined; + const merges = (await git(checkout, ['rev-list', '--count', '--merges', `${target.commit}..${head}`], signal)).trim(); + if (merges === '0') { + const cherry = (await git(checkout, ['cherry', target.commit, head], signal, GIT_TIMEOUT_MS, GIT_LIST_LIMIT)) + .split('\n') + .filter(Boolean); + if (cherry.every((line) => line.startsWith('- '))) return `patch-equivalent to ${target.ref}`; + } + let base: string; + try { + base = (await git(checkout, ['merge-base', target.commit, head], signal)).trim(); + } catch { + signal?.throwIfAborted(); + return undefined; + } + const diff = (from: string): Promise => + git( + checkout, + ['diff', '--name-only', '-z', '--no-renames', '--no-ext-diff', '--no-textconv', from, head], + signal, + GIT_TIMEOUT_MS, + GIT_LIST_LIMIT, + ); + const touched = pathList(await diff(base)); + const differing = new Set(pathList(await diff(target.commit))); + return touched.every((path) => !differing.has(path)) ? `same content as ${target.ref}` : undefined; +} + +function errorDetail(error: unknown): string { + const stderr = (error as { stderr?: unknown }).stderr; + const text = typeof stderr === 'string' && stderr.trim() ? stderr : error instanceof Error ? error.message : String(error); + return text.trim().split('\n')[0]!.slice(0, 200); +} + +/** Absent is the only answer that clears a check; anything unreadable counts as present. */ +async function present(path: string): Promise { + try { + await lstat(path); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code !== 'ENOENT'; + } +} + +async function modifiedAt(path: string): Promise { + try { + return (await lstat(path)).mtimeMs; + } catch { + return 0; + } +} + +/** The newest on-disk sign of use: the worktree directory itself and its Git metadata. */ +async function lastActivity(root: string, metadata: string): Promise { + const times = await Promise.all( + [root, metadata, ...ACTIVITY_STATE.map((path) => join(metadata, path))].map(modifiedAt), + ); + return Math.max(...times); +} + +async function worktreeNames(checkout: string): Promise<{ names: string[]; truncated: boolean }> { + let directory; + try { + directory = await opendir(join(checkout, LINKED_WORKTREE_DIRECTORY)); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return { names: [], truncated: false }; + throw error; + } + const names: string[] = []; + try { + for await (const entry of directory) { + if (names.length >= SCAN_LIMIT) return { names, truncated: true }; + if (isValidLinkedWorktreeName(entry.name)) names.push(entry.name); + } + } finally { + await directory.close().catch(() => undefined); + } + return { names, truncated: false }; +} + +async function freeSpace(paths: readonly string[]): Promise> { + const space = new Map(); + for (const path of paths) { + try { + const device = String((await lstat(path)).dev); + if (space.has(device)) continue; + const status = await statfs(path, { bigint: true }); + space.set(device, Number(status.bavail * status.bsize)); + } catch { + // Space reporting is informational; it never decides a retirement. + } + } + return space; +} + +function describeChanges(status: string): string { + const entries = status.split('\0').filter(Boolean); + const untracked = entries.filter((entry) => entry.startsWith('?? ')).length; + return untracked === entries.length + ? 'untracked files' + : untracked > 0 + ? 'tracked changes and untracked files' + : 'tracked changes'; +} + +function formatBytes(bytes: number): string { + const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB']; + let value = bytes; + let unit = 0; + while (value >= 1024 && unit < units.length - 1) { + value /= 1024; + unit += 1; + } + return `${unit === 0 ? value : value.toFixed(1)} ${units[unit]}`; +} + +export function describeWorktreeRetirement(summary: WorktreeRetirementSummary): string { + const kept = Object.entries(summary.kept) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([reason, count]) => `${reason} ${count}`); + const keptTotal = Object.values(summary.kept).reduce((total, count) => total + (count ?? 0), 0); + return [ + `worktree retirement: retired ${summary.retired.length}`, + `kept ${keptTotal}${kept.length ? ` (${kept.join(', ')})` : ''}`, + ...(summary.freedBytes > 0 ? [`freed about ${formatBytes(summary.freedBytes)}`] : []), + ...(summary.truncated ? ['more next pass'] : []), + ].join(', '); +} + +/** Everything short of the worker's own activity that could make removal lose work. */ +async function inspect( + candidate: Candidate, + options: WorktreeRetirementOptions, + signal?: AbortSignal, +): Promise { + const { lane, metadata, name, source } = candidate; + try { + if (await options.isQuarantined?.(source.workspaceId, name)) return { kept: 'quarantined' }; + } catch { + return { kept: 'quarantined', detail: 'quarantine state is unreadable' }; + } + if (await present(join(metadata, 'locked'))) return { kept: 'locked' }; + for (const state of OPERATION_STATE) { + if (await present(join(metadata, state))) return { kept: 'operation', detail: state }; + } + let head: string; + try { + head = (await git(lane.root, ['rev-parse', '--verify', '--quiet', 'HEAD^{commit}'], signal)).trim(); + } catch { + signal?.throwIfAborted(); + return { kept: 'no-commit' }; + } + if (!/^[0-9a-f]{40,64}$/.test(head)) return { kept: 'no-commit' }; + let status: string; + try { + status = await git( + lane.root, + ['status', '--porcelain', '-z', '--untracked-files=normal', '--ignore-submodules=none'], + signal, + ); + } catch (error) { + signal?.throwIfAborted(); + if ((error as NodeJS.ErrnoException).code === 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER') { + return { kept: 'dirty', detail: 'too many changes to list' }; + } + return { kept: 'failed', detail: `status: ${errorDetail(error)}` }; + } + if (status.length > 0) return { kept: 'dirty', detail: describeChanges(status) }; + let remote: string; + try { + remote = ( + await git( + lane.checkoutRoot, + ['for-each-ref', '--count=1', '--contains', head, '--format=%(refname)', 'refs/remotes/'], + signal, + ) + ).trim(); + } catch (error) { + signal?.throwIfAborted(); + return { kept: 'failed', detail: `remote containment: ${errorDetail(error)}` }; + } + if (remote) return { head, basis: `contained in ${remote.replace(/^refs\/remotes\//, '')}` }; + let merged: string | undefined; + try { + merged = await mergedByContent(lane, head, signal); + } catch (error) { + signal?.throwIfAborted(); + return { kept: 'failed', detail: `merged content: ${errorDetail(error)}` }; + } + return merged + ? { head, basis: merged } + : { kept: 'unpushed', detail: 'not on a remote-tracking ref or merged into the default branch' }; +} + +/** Re-checked under the lane reservation: anything that moved since inspection defers to the next pass. */ +async function remove( + candidate: Candidate, + head: string, + activity: LinkedWorktreeActivity | undefined, + signal?: AbortSignal, +): Promise { + const { lane, metadata, name, source } = candidate; + if (activity?.lastUsed(source.workspaceId, name) !== candidate.usedAt) { + return { kept: 'changed', detail: 'used during inspection' }; + } + let current: VerifiedLinkedWorktree; + try { + current = await verifyLinkedWorktree(source.root, name, source.identity); + } catch { + return { kept: 'changed', detail: 'no longer a verified linked worktree' }; + } + if (current.identity.dev !== lane.identity.dev || current.identity.ino !== lane.identity.ino) { + return { kept: 'changed', detail: 'worktree directory was replaced' }; + } + if ((await lastActivity(lane.root, metadata)) !== candidate.diskActiveAt) { + return { kept: 'changed', detail: 'used during inspection' }; + } + if (await present(join(metadata, 'locked'))) return { kept: 'locked' }; + try { + const now = (await git(lane.root, ['rev-parse', '--verify', '--quiet', 'HEAD^{commit}'], signal)).trim(); + if (now !== head) return { kept: 'changed', detail: 'HEAD moved during inspection' }; + } catch { + signal?.throwIfAborted(); + return { kept: 'changed', detail: 'HEAD is unreadable' }; + } + signal?.throwIfAborted(); + try { + // Never --force: Git re-checks for changes and untracked files itself and refuses a dirty or locked worktree. + await git(lane.checkoutRoot, ['worktree', 'remove', lane.root], undefined, GIT_REMOVE_TIMEOUT_MS); + } catch (error) { + return { kept: 'failed', detail: `remove: ${errorDetail(error)}` }; + } + return 'retired'; +} + +/** + * Retire linked worktrees beneath each checkout's `.worktrees` that are + * verified, idle, clean, free of in-progress operations and locks, and whose + * HEAD is contained in a remote-tracking ref or already merged into the + * remote's default branch by content. Removal is `git worktree remove` + * without `--force`; branches are kept, so `git worktree add` restores any of + * them. Each worktree is judged independently; one failure never ends the pass. + */ +export async function retireStaleWorktrees( + options: WorktreeRetirementOptions, + signal?: AbortSignal, +): Promise { + const now = Date.now(); + const idleMs = options.idleMs ?? DEFAULT_WORKTREE_IDLE_DAYS * DAY_MS; + const summary: WorktreeRetirementSummary = { retired: [], kept: {}, freedBytes: 0, truncated: false }; + const keep = (source: WorktreeRetirementSource, name: string, reason: WorktreeKeptReason, detail?: string): void => { + summary.kept[reason] = (summary.kept[reason] ?? 0) + 1; + options.log?.('debug', `worktree retirement kept ${source.workspaceId}:${name}: ${reason}${detail ? ` (${detail})` : ''}`); + }; + const candidates: Candidate[] = []; + for (const source of options.sources) { + signal?.throwIfAborted(); + try { + if (await options.isQuarantined?.(source.workspaceId)) { + options.log?.('debug', `worktree retirement skipped ${source.workspaceId}: checkout is quarantined`); + continue; + } + const { names, truncated } = await worktreeNames(source.root); + if (truncated) { + options.log?.('debug', `worktree retirement read only the first ${SCAN_LIMIT} entries of ${source.workspaceId}`); + } + for (const name of names) { + signal?.throwIfAborted(); + let lane: VerifiedLinkedWorktree; + try { + lane = await verifyLinkedWorktree(source.root, name, source.identity); + } catch (error) { + keep(source, name, 'unverified', errorDetail(error)); + continue; + } + const metadata = join(lane.commonGitDir, 'worktrees', name); + const diskActiveAt = await lastActivity(lane.root, metadata); + const usedAt = options.activity?.lastUsed(source.workspaceId, name); + const activeAt = Math.max(diskActiveAt, usedAt ?? 0); + if (now - activeAt < idleMs) { + keep(source, name, 'recent'); + continue; + } + candidates.push({ source, name, lane, metadata, diskActiveAt, usedAt, activeAt }); + } + } catch (error) { + signal?.throwIfAborted(); + options.log?.('debug', `worktree retirement skipped ${source.workspaceId}: ${errorDetail(error)}`); + } + } + if (candidates.length === 0) { + options.rotation?.clear(); + return summary; + } + const { rotation } = options; + const key = (candidate: Candidate): string => `${candidate.source.workspaceId}\0${candidate.name}`; + if (rotation) { + const current = new Set(candidates.map(key)); + for (const entry of rotation) { + if (!current.has(entry)) rotation.delete(entry); + } + if (rotation.size === current.size) rotation.clear(); + } + const visited = (candidate: Candidate): number => (rotation?.has(key(candidate)) ? 1 : 0); + candidates.sort((a, b) => visited(a) - visited(b) || a.activeAt - b.activeAt); + const checkouts = [...new Set(candidates.map((candidate) => candidate.lane.checkoutRoot))]; + const before = await freeSpace(checkouts); + const retiredCheckouts = new Set(); + let inspected = 0; + for (const candidate of candidates) { + signal?.throwIfAborted(); + const { source, name } = candidate; + if ( + inspected >= (options.limits?.inspect ?? INSPECT_LIMIT) || + summary.retired.length >= (options.limits?.retire ?? RETIRE_LIMIT) + ) { + summary.truncated ||= visited(candidate) === 0; + keep(source, name, 'deferred'); + continue; + } + inspected += 1; + rotation?.add(key(candidate)); + try { + const verdict = await inspect(candidate, options, signal); + if ('kept' in verdict) { + keep(source, name, verdict.kept, verdict.detail); + continue; + } + const retire = (): Promise => remove(candidate, verdict.head, options.activity, signal); + const reserved = options.activity + ? await options.activity.whileIdle(source.workspaceId, name, retire) + : { ran: true as const, value: await retire() }; + if (!reserved.ran) { + keep(source, name, 'active'); + continue; + } + if (reserved.value !== 'retired') { + keep(source, name, reserved.value.kept, reserved.value.detail); + continue; + } + summary.retired.push(`${source.workspaceId}:${name}`); + retiredCheckouts.add(candidate.lane.checkoutRoot); + options.log?.('debug', `worktree retirement retired ${source.workspaceId}:${name} (${verdict.basis}); its branch is kept`); + } catch (error) { + signal?.throwIfAborted(); + keep(source, name, 'failed', errorDetail(error)); + } + } + for (const checkout of retiredCheckouts) { + try { + await git(checkout, ['worktree', 'prune']); + } catch (error) { + options.log?.('debug', `worktree retirement could not prune worktree metadata: ${errorDetail(error)}`); + } + } + if (retiredCheckouts.size > 0) { + const after = await freeSpace(checkouts); + for (const [device, free] of after) { + summary.freedBytes += Math.max(0, free - (before.get(device) ?? free)); + } + } + return summary; +} + +export interface WorktreeRetirementSchedulerOptions extends WorktreeRetirementOptions { + intervalMs?: number; + startDelayMs?: number; + continuationMs?: number; +} + +/** + * Runs retirement passes in the background: once shortly after start, then + * every interval, sooner while a backlog remains. Passes never overlap; a pass + * requested while one runs joins it. + */ +export class WorktreeRetirementScheduler { + private readonly controller = new AbortController(); + private readonly rotation = new Set(); + private running?: Promise; + private timer?: NodeJS.Timeout; + private started = false; + + constructor(private readonly options: WorktreeRetirementSchedulerOptions) {} + + start(): void { + if (this.started || this.controller.signal.aborted) return; + this.started = true; + if (this.running == null) this.schedule(this.options.startDelayMs ?? WORKTREE_RETIREMENT_START_DELAY_MS); + } + + /** Run a pass now, or join the one already running. */ + runNow(): Promise { + if (this.controller.signal.aborted) return Promise.resolve(undefined); + this.running ??= this.pass().finally(() => { + this.running = undefined; + }); + return this.running; + } + + /** Stop scheduling; a removal already under way finishes first. */ + async stop(): Promise { + this.controller.abort(); + clearTimeout(this.timer); + await this.running; + } + + private schedule(delayMs: number): void { + clearTimeout(this.timer); + if (!this.started || this.controller.signal.aborted) return; + this.timer = setTimeout(() => void this.runNow(), delayMs); + this.timer.unref(); + } + + private async pass(): Promise { + clearTimeout(this.timer); + let summary: WorktreeRetirementSummary | undefined; + try { + summary = await retireStaleWorktrees( + { ...this.options, rotation: this.rotation }, + this.controller.signal, + ); + this.options.log?.('info', describeWorktreeRetirement(summary)); + } catch (error) { + if (!this.controller.signal.aborted) { + this.options.log?.('info', `worktree retirement pass failed: ${errorDetail(error)}`); + } + } + this.schedule( + summary?.truncated + ? (this.options.continuationMs ?? WORKTREE_RETIREMENT_CONTINUATION_MS) + : (this.options.intervalMs ?? WORKTREE_RETIREMENT_INTERVAL_MS), + ); + return summary; + } +} From 82fa757a285bf57feaf229466e63b91bfef89ffb Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Fri, 2 Oct 2026 21:37:12 -0400 Subject: [PATCH 2/5] fix: Fence Checkout Requests During Retirement and Never Cut Removal Short - Checkout requests wait for an in-progress retirement of any lane beneath them, since root commands can reach .worktrees/*. - Requests waiting on a retirement honor their own cancellation. - git worktree remove runs without a timeout; a killed removal could leave a half-deleted worktree that no later pass can verify. - Drop the repository-wide git worktree prune: remove already deletes the retired worktree's metadata, and prune would also expire other registered worktrees that are only temporarily unavailable. - Read every .worktrees entry each pass instead of the first 4096. --- packages/code/README.md | 22 ++++--- packages/code/src/linked-worktrees.ts | 62 ++++++++++++++----- packages/code/src/worktree-retirement.test.ts | 29 ++++++++- packages/code/src/worktree-retirement.ts | 32 +++++----- 4 files changed, 105 insertions(+), 40 deletions(-) diff --git a/packages/code/README.md b/packages/code/README.md index 3928abf7..c19612a2 100644 --- a/packages/code/README.md +++ b/packages/code/README.md @@ -955,7 +955,8 @@ keeps its own dependencies and build output. A worker with lanes therefore retires stale ones itself, with no configuration: a pass runs two minutes after startup and every six hours after that, sooner while a backlog remains, and also before managed environment setup would be deferred for low disk space. -Passes run in the background, never overlap, and never hold back requests. +Passes run in the background and never overlap. Only an actual removal holds +back requests, and only those for that lane or its checkout. A worktree in a writable registered root is retired only when **all** of these hold; otherwise it is kept and the reason is counted: @@ -983,13 +984,18 @@ hold; otherwise it is kept and the reason is counted: Commits beyond a live upstream are never treated as merged. Removal is `git worktree remove` **without** `--force`, so Git re-checks for -changes itself, followed by `git worktree prune`. Ignored files such as -`node_modules`, build output and ignored `.env` files go with the worktree; -that is the space being reclaimed. The branch is kept, so -`git worktree add .worktrees/ ` restores the worktree. A lane -request that arrives during removal waits for it and then fails as an unknown -worktree. Each pass inspects at most 128 idle worktrees and removes at most 32, -oldest first. Each pass logs one summary line, for example +changes itself and deletes only that worktree's metadata. No repository-wide +`git worktree prune` runs, so other registered worktrees that are temporarily +unavailable stay registered. Removal has no timeout, because a half-deleted +worktree could no longer be recognized. Ignored files such as `node_modules`, +build output and ignored `.env` files go with the worktree; that is the space +being reclaimed. The branch is kept, so +`git worktree add .worktrees/ ` restores the worktree. Lane and +checkout requests that arrive during a removal wait for it, or for their own +cancellation; a lane request then fails as an unknown worktree. Each pass reads +every `.worktrees` entry, inspects at most 128 idle worktrees and removes at +most 32, oldest first, rotating so that worktrees kept for lasting reasons +cannot hide the rest. Each pass logs one summary line, for example `worktree retirement: retired 3, kept 12 (dirty 2, recent 8, unpushed 2), freed about 4.1 GiB`; set `LIBRECHAT_CODE_LOG_LEVEL=debug` to log every kept worktree and its reason. diff --git a/packages/code/src/linked-worktrees.ts b/packages/code/src/linked-worktrees.ts index 014b9787..08cd0947 100644 --- a/packages/code/src/linked-worktrees.ts +++ b/packages/code/src/linked-worktrees.ts @@ -41,8 +41,9 @@ export interface LinkedWorktreeActivity { lastUsed(workspaceId: string, worktree: string): number | undefined; /** * Run `task` only while neither the lane nor its checkout has a request in - * flight. Lane requests that arrive meanwhile wait for it to finish, then - * verify the lane again; checkout requests are never held back. + * flight. Requests for the lane or its checkout that arrive meanwhile wait + * for it to finish (or for their own cancellation); a lane request then + * verifies the lane again. */ whileIdle( workspaceId: string, @@ -240,8 +241,8 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link private readonly checkoutRequests = new Map(); /** Last request start or finish by lane internal ID, oldest first. */ private readonly used = new Map(); - /** Lanes being retired; their requests wait for retirement to settle. */ - private readonly retiring = new Map>(); + /** Lanes being retired by internal ID; requests for the lane or its checkout wait for them to settle. */ + private readonly retiring = new Map }>(); constructor(private readonly options: LinkedWorktreeWorkspaceToolsOptions) { this.mutationFailuresAreAtomic = options.delegate.mutationFailuresAreAtomic; @@ -381,7 +382,8 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link return { ran: false }; } let settle!: () => void; - this.retiring.set(internalId, new Promise((resolve) => { settle = resolve; })); + const done = new Promise((resolve) => { settle = resolve; }); + this.retiring.set(internalId, { workspaceId, done }); try { return { ran: true, value: await task() }; } finally { @@ -418,21 +420,52 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link } } - private async inLane(workspaceId: string, worktree: string, task: () => Promise): Promise { + /** Wait out retirements a request could collide with, unless the request is cancelled first. */ + private async afterRetirement( + collides: (internalId: string, workspaceId: string) => boolean, + signal: AbortSignal | undefined, + ): Promise { + const pending = [...this.retiring] + .filter(([internalId, entry]) => collides(internalId, entry.workspaceId)) + .map(([, entry]) => entry.done); + if (pending.length === 0) return; + signal?.throwIfAborted(); + await new Promise((resolve, reject) => { + const onAbort = (): void => reject(signal?.reason); + signal?.addEventListener('abort', onAbort, { once: true }); + void Promise.all(pending).then(() => { + signal?.removeEventListener('abort', onAbort); + resolve(); + }); + }); + } + + private async inCheckout(workspaceId: string, task: () => Promise, signal?: AbortSignal): Promise { + return await this.tracked(this.checkoutRequests, workspaceId, async () => { + await this.afterRetirement((_internalId, laneWorkspaceId) => laneWorkspaceId === workspaceId, signal); + return await task(); + }, false); + } + + private async inLane( + workspaceId: string, + worktree: string, + task: () => Promise, + signal?: AbortSignal, + ): Promise { const internalId = linkedWorktreeWorkspaceId(workspaceId, worktree); return await this.tracked(this.laneRequests, internalId, async () => { - await this.retiring.get(internalId); + await this.afterRetirement((retiringId) => retiringId === internalId, signal); return await task(); }, true); } async execute(request: WorkspaceToolRequest, signal?: AbortSignal): Promise { if (request.worktree == null) { - return await this.tracked( - this.checkoutRequests, + return await this.inCheckout( request.workspaceId, () => this.options.delegate.execute(request, signal), - false, + signal, ); } if (request.operation === 'execute_command' && request.environmentAction) { @@ -455,7 +488,7 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link } const executor = await this.fileExecutor(internalId, lane, source); return publicResult(await executor.execute(laneRequest, signal), request.workspaceId); - }); + }, signal); } async executeProgrammatic( @@ -469,11 +502,10 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link if (!delegate) { throw new WorkspaceToolError('Workspace programmatic execution is unavailable', 'COMMAND_DISABLED'); } - return await this.tracked( - this.checkoutRequests, + return await this.inCheckout( workspaceId, () => delegate.executeProgrammatic(workspaceId, request, signal), - false, + signal, ); } return await this.inLane(workspaceId, worktree, async () => { @@ -485,6 +517,6 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link const pool = await this.registerCommandRoot(internalId, lane, source); const { workspace_worktree: _worktree, ...body } = request.body; return await pool.executeProgrammatic(internalId, { ...request, body }, signal); - }); + }, signal); } } diff --git a/packages/code/src/worktree-retirement.test.ts b/packages/code/src/worktree-retirement.test.ts index 427e7e20..76a5d6ee 100644 --- a/packages/code/src/worktree-retirement.test.ts +++ b/packages/code/src/worktree-retirement.test.ts @@ -1,6 +1,6 @@ import assert from 'node:assert/strict'; import { execFile } from 'node:child_process'; -import { mkdir, mkdtemp, readFile, realpath, rm, stat, utimes, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, realpath, rename, rm, stat, utimes, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { promisify } from 'node:util'; @@ -357,9 +357,17 @@ test('a lane request that arrives during retirement waits, then finds the lane g let settled = false; const request = tools.execute(laneCommand('leaving')); request.catch(() => undefined).finally(() => { settled = true; }); + let checkoutSettled = false; + const checkout = tools.execute({ protocolVersion: 1, operation: 'read_file', workspaceId: 'repo', path: 'README.md' }); + checkout.finally(() => { checkoutSettled = true; }); + const cancelled = new AbortController(); + const abandoned = tools.execute(laneCommand('leaving'), cancelled.signal); await pause(20); assert.equal(settled, false, 'the lane request waits for retirement'); + assert.equal(checkoutSettled, false, 'a checkout request waits too: it can reach the lane being removed'); assert.deepEqual(await tools.whileIdle('repo', 'leaving', async () => 'again'), { ran: false }); + cancelled.abort(new Error('caller gave up')); + await assert.rejects(abandoned, /caller gave up/, 'cancellation ends the wait without the retirement'); release(); assert.deepEqual(await retiring, { ran: true, value: 'removed' }); @@ -367,9 +375,28 @@ test('a lane request that arrives during retirement waits, then finds the lane g request, (error: unknown) => error instanceof WorkspaceToolError && error.code === 'INVALID_REQUEST', ); + const read = await checkout; + assert.equal(read.operation === 'read_file' && read.content.trimEnd(), 'root'); assert.equal(executions, 0); }); +test('retirement never prunes other registered worktrees that are temporarily missing', async (t) => { + const root = await repository(t); + const external = join(root, '..', 'external'); + await git(root, 'worktree', 'add', '-q', '-b', 'external', external); + await rename(external, `${external}.unmounted`); + await worktree(root, 'done'); + await age(root, 'done'); + + const summary = await retireStaleWorktrees({ sources: sources(root) }); + + assert.deepEqual(summary.retired, ['repo:done']); + assert.ok(await exists(join(root, '.git', 'worktrees', 'external')), 'the unavailable worktree stays registered'); + assert.equal(await exists(join(root, '.git', 'worktrees', 'done')), false, 'removal deletes its own metadata'); + await rename(`${external}.unmounted`, external); + assert.equal(await git(external, 'branch', '--show-current'), 'external'); +}); + test('a checkout request in flight defers retirement of its lanes', async (t) => { const root = await repository(t); const path = await worktree(root, 'quiet'); diff --git a/packages/code/src/worktree-retirement.ts b/packages/code/src/worktree-retirement.ts index d13cb3d8..86248c56 100644 --- a/packages/code/src/worktree-retirement.ts +++ b/packages/code/src/worktree-retirement.ts @@ -18,19 +18,23 @@ export const WORKTREE_RETIREMENT_INTERVAL_MS = 6 * 60 * 60 * 1000; const WORKTREE_RETIREMENT_START_DELAY_MS = 2 * 60 * 1000; /** A pass that hit a bound continues soon instead of waiting a full interval. */ const WORKTREE_RETIREMENT_CONTINUATION_MS = 5 * 60 * 1000; -/** Directory entries read beneath one checkout's `.worktrees` per pass. */ -const SCAN_LIMIT = 4096; +/** + * Directory entries read beneath one checkout's `.worktrees` per pass: a + * memory bound far above any real checkout, so every pass sees every entry. + * Git inspection is what is bounded per pass, and rotation keeps it fair. + */ +const SCAN_LIMIT = 65_536; /** Idle worktrees whose Git state is inspected per pass, oldest first. */ const INSPECT_LIMIT = 128; /** Worktrees removed per pass. */ const RETIRE_LIMIT = 32; const GIT_TIMEOUT_MS = 30_000; /** - * Removal deletes ignored build output too, which can take a while. It is never - * cut short: a half-deleted worktree loses its `.git` file and can no longer be - * verified, retired or recognized. + * Removal deletes ignored build output too, which can take a while. It has no + * timeout and ignores cancellation: a half-deleted worktree loses its `.git` + * file and can no longer be verified, retired or recognized. */ -const GIT_REMOVE_TIMEOUT_MS = 30 * 60_000; +const GIT_REMOVE_NO_TIMEOUT = 0; const GIT_OUTPUT_LIMIT = 64 * 1024; /** Path and commit lists between a stale branch and its default branch can be long. */ const GIT_LIST_LIMIT = 16 * 1024 * 1024; @@ -470,7 +474,7 @@ async function remove( signal?.throwIfAborted(); try { // Never --force: Git re-checks for changes and untracked files itself and refuses a dirty or locked worktree. - await git(lane.checkoutRoot, ['worktree', 'remove', lane.root], undefined, GIT_REMOVE_TIMEOUT_MS); + await git(lane.checkoutRoot, ['worktree', 'remove', lane.root], undefined, GIT_REMOVE_NO_TIMEOUT); } catch (error) { return { kept: 'failed', detail: `remove: ${errorDetail(error)}` }; } @@ -482,8 +486,11 @@ async function remove( * verified, idle, clean, free of in-progress operations and locks, and whose * HEAD is contained in a remote-tracking ref or already merged into the * remote's default branch by content. Removal is `git worktree remove` - * without `--force`; branches are kept, so `git worktree add` restores any of - * them. Each worktree is judged independently; one failure never ends the pass. + * without `--force`, which also deletes that worktree's own metadata; no + * repository-wide `git worktree prune` runs, since it would also expire other + * registered worktrees that are only temporarily unavailable. Branches are + * kept, so `git worktree add` restores any of them. Each worktree is judged + * independently; one failure never ends the pass. */ export async function retireStaleWorktrees( options: WorktreeRetirementOptions, @@ -590,13 +597,6 @@ export async function retireStaleWorktrees( keep(source, name, 'failed', errorDetail(error)); } } - for (const checkout of retiredCheckouts) { - try { - await git(checkout, ['worktree', 'prune']); - } catch (error) { - options.log?.('debug', `worktree retirement could not prune worktree metadata: ${errorDetail(error)}`); - } - } if (retiredCheckouts.size > 0) { const after = await freeSpace(checkouts); for (const [device, free] of after) { From bf294eddf3feb72f8c431ac619719d5395328620 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Fri, 2 Oct 2026 21:46:18 -0400 Subject: [PATCH 3/5] fix: Recheck Quarantine Under the Lane Reservation A checkout or lane request that finished between inspection and the lane reservation may have left an uncertain mutation quarantined. Re-read both quarantine markers inside the reservation, immediately before removal. --- packages/code/src/worktree-retirement.test.ts | 21 +++++++++++++++++++ packages/code/src/worktree-retirement.ts | 17 ++++++++++++--- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/packages/code/src/worktree-retirement.test.ts b/packages/code/src/worktree-retirement.test.ts index 76a5d6ee..5e31f5b5 100644 --- a/packages/code/src/worktree-retirement.test.ts +++ b/packages/code/src/worktree-retirement.test.ts @@ -455,6 +455,27 @@ test('quarantined lanes and checkouts are left for the operator', async (t) => { assert.ok(await exists(join(root, '.worktrees', 'held'))); }); +test('a quarantine that appears after inspection is caught under the lane reservation', async (t) => { + const root = await repository(t); + await worktree(root, 'late'); + await age(root, 'late'); + for (const quarantinedLater of [undefined, 'late']) { + const asked: string[] = []; + const summary = await retireStaleWorktrees({ + sources: sources(root), + activity: await laneTools(root), + isQuarantined: async (_workspaceId, worktree) => { + asked.push(worktree ?? ''); + // Clear for the scan and inspection; quarantined once the lane is reserved. + return asked.length > 2 && worktree === quarantinedLater; + }, + }); + assert.deepEqual(summary.kept, { quarantined: 1 }); + assert.deepEqual(asked, ['', 'late', '', ...(quarantinedLater ? ['late'] : [])]); + assert.ok(await exists(join(root, '.worktrees', 'late'))); + } +}); + test('one worktree failing does not stop the pass', async (t) => { const root = await repository(t); await worktree(root, 'broken'); diff --git a/packages/code/src/worktree-retirement.ts b/packages/code/src/worktree-retirement.ts index 86248c56..64ae6b87 100644 --- a/packages/code/src/worktree-retirement.ts +++ b/packages/code/src/worktree-retirement.ts @@ -444,13 +444,24 @@ async function inspect( async function remove( candidate: Candidate, head: string, - activity: LinkedWorktreeActivity | undefined, + options: WorktreeRetirementOptions, signal?: AbortSignal, ): Promise { const { lane, metadata, name, source } = candidate; - if (activity?.lastUsed(source.workspaceId, name) !== candidate.usedAt) { + if (options.activity?.lastUsed(source.workspaceId, name) !== candidate.usedAt) { return { kept: 'changed', detail: 'used during inspection' }; } + // A request that finished between inspection and this reservation may have quarantined either. + try { + if ( + (await options.isQuarantined?.(source.workspaceId)) || + (await options.isQuarantined?.(source.workspaceId, name)) + ) { + return { kept: 'quarantined' }; + } + } catch { + return { kept: 'quarantined', detail: 'quarantine state is unreadable' }; + } let current: VerifiedLinkedWorktree; try { current = await verifyLinkedWorktree(source.root, name, source.identity); @@ -577,7 +588,7 @@ export async function retireStaleWorktrees( keep(source, name, verdict.kept, verdict.detail); continue; } - const retire = (): Promise => remove(candidate, verdict.head, options.activity, signal); + const retire = (): Promise => remove(candidate, verdict.head, options, signal); const reserved = options.activity ? await options.activity.whileIdle(source.workspaceId, name, retire) : { ran: true as const, value: await retire() }; From 1563f372896526c345756c75a864378f19ab11e6 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Fri, 2 Oct 2026 21:55:01 -0400 Subject: [PATCH 4/5] fix: Defer Retirement When the Checkout Was Used During Inspection A checkout request can write ignored files inside .worktrees/ that git status never reports, and finish before the lane reservation. Count checkout request starts and finishes, capture the count before inspecting each worktree, and keep the worktree for a later pass when it changed by the time the reservation is held. --- packages/code/README.md | 3 ++- packages/code/src/linked-worktrees.ts | 16 +++++++++++-- packages/code/src/worktree-retirement.test.ts | 24 +++++++++++++++++++ packages/code/src/worktree-retirement.ts | 7 ++++++ 4 files changed, 47 insertions(+), 3 deletions(-) diff --git a/packages/code/README.md b/packages/code/README.md index c19612a2..55f5c668 100644 --- a/packages/code/README.md +++ b/packages/code/README.md @@ -964,7 +964,8 @@ hold; otherwise it is kept and the reason is counted: - It verifies as a linked worktree of the checkout under `.worktrees/`, as for lane admission. The checkout itself and worktrees elsewhere are never touched. -- Neither the lane nor its checkout has a request in flight, and both this +- Neither the lane nor its checkout has a request in flight or ran one while + the worktree was being inspected, and both this worker's last use of the lane and the newest on-disk activity (the worktree directory and its Git `HEAD`, `index`, `logs/HEAD`, `ORIG_HEAD` and `FETCH_HEAD`) are older than the idle threshold, seven days by default. diff --git a/packages/code/src/linked-worktrees.ts b/packages/code/src/linked-worktrees.ts index 08cd0947..225abebd 100644 --- a/packages/code/src/linked-worktrees.ts +++ b/packages/code/src/linked-worktrees.ts @@ -39,6 +39,8 @@ const LINKED_WORKTREE_ACTIVITY_LIMIT = 4096; export interface LinkedWorktreeActivity { /** When a request for the lane last started or finished in this process. */ lastUsed(workspaceId: string, worktree: string): number | undefined; + /** Changes whenever a checkout request (outside any lane) starts or finishes. */ + checkoutActivity(workspaceId: string): number; /** * Run `task` only while neither the lane nor its checkout has a request in * flight. Requests for the lane or its checkout that arrive meanwhile wait @@ -241,6 +243,8 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link private readonly checkoutRequests = new Map(); /** Last request start or finish by lane internal ID, oldest first. */ private readonly used = new Map(); + /** Checkout request starts and finishes by workspace ID. */ + private readonly checkoutEvents = new Map(); /** Lanes being retired by internal ID; requests for the lane or its checkout wait for them to settle. */ private readonly retiring = new Map }>(); @@ -368,6 +372,10 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link return this.used.get(linkedWorktreeWorkspaceId(workspaceId, worktree)); } + checkoutActivity(workspaceId: string): number { + return this.checkoutEvents.get(workspaceId) ?? 0; + } + async whileIdle( workspaceId: string, worktree: string, @@ -408,15 +416,19 @@ export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor, Link task: () => Promise, lane: boolean, ): Promise { + const record = (): void => { + if (lane) this.touch(key); + else this.checkoutEvents.set(key, (this.checkoutEvents.get(key) ?? 0) + 1); + }; requests.set(key, (requests.get(key) ?? 0) + 1); - if (lane) this.touch(key); + record(); try { return await task(); } finally { const remaining = (requests.get(key) ?? 1) - 1; if (remaining > 0) requests.set(key, remaining); else requests.delete(key); - if (lane) this.touch(key); + record(); } } diff --git a/packages/code/src/worktree-retirement.test.ts b/packages/code/src/worktree-retirement.test.ts index 5e31f5b5..4c81b2e5 100644 --- a/packages/code/src/worktree-retirement.test.ts +++ b/packages/code/src/worktree-retirement.test.ts @@ -433,6 +433,30 @@ test('a checkout request in flight defers retirement of its lanes', async (t) => assert.deepEqual(after.retired, ['repo:quiet']); }); +test('a checkout request that runs during inspection defers retirement', async (t) => { + const root = await repository(t); + await worktree(root, 'touched'); + await age(root, 'touched'); + const tools = await laneTools(root); + const before = tools.checkoutActivity('repo'); + await tools.execute({ protocolVersion: 1, operation: 'read_file', workspaceId: 'repo', path: 'README.md' }); + assert.equal(tools.checkoutActivity('repo'), before + 2, 'a checkout request counts at start and finish'); + + let reads = 0; + const duringInspection = { + lastUsed: (workspaceId: string, name: string) => tools.lastUsed(workspaceId, name), + // The second read, under the reservation, sees a checkout request that ran during inspection. + checkoutActivity: () => reads++, + whileIdle: (workspaceId: string, name: string, task: () => Promise) => tools.whileIdle(workspaceId, name, task), + }; + const deferred = await retireStaleWorktrees({ sources: sources(root), activity: duringInspection }); + assert.deepEqual(deferred.kept, { changed: 1 }); + assert.ok(await exists(join(root, '.worktrees', 'touched'))); + + const quiet = await retireStaleWorktrees({ sources: sources(root), activity: tools }); + assert.deepEqual(quiet.retired, ['repo:touched']); +}); + test('quarantined lanes and checkouts are left for the operator', async (t) => { const root = await repository(t); await worktree(root, 'held'); diff --git a/packages/code/src/worktree-retirement.ts b/packages/code/src/worktree-retirement.ts index 64ae6b87..39a7b60c 100644 --- a/packages/code/src/worktree-retirement.ts +++ b/packages/code/src/worktree-retirement.ts @@ -114,6 +114,8 @@ interface Candidate { diskActiveAt: number; /** This worker's own last use of the lane, re-read under the reservation. */ usedAt?: number; + /** Checkout request activity when inspection began, re-read under the reservation. */ + checkoutAt?: number; /** Newest of on-disk activity and this worker's own lane use. */ activeAt: number; } @@ -451,6 +453,10 @@ async function remove( if (options.activity?.lastUsed(source.workspaceId, name) !== candidate.usedAt) { return { kept: 'changed', detail: 'used during inspection' }; } + // A checkout request can reach `.worktrees/*`, including ignored files that `git status` never reports. + if (options.activity?.checkoutActivity(source.workspaceId) !== candidate.checkoutAt) { + return { kept: 'changed', detail: 'checkout used during inspection' }; + } // A request that finished between inspection and this reservation may have quarantined either. try { if ( @@ -583,6 +589,7 @@ export async function retireStaleWorktrees( inspected += 1; rotation?.add(key(candidate)); try { + candidate.checkoutAt = options.activity?.checkoutActivity(source.workspaceId); const verdict = await inspect(candidate, options, signal); if ('kept' in verdict) { keep(source, name, verdict.kept, verdict.detail); From 7c8108f7390260f5473a7075f388a545784ce8b0 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Fri, 2 Oct 2026 21:58:54 -0400 Subject: [PATCH 5/5] test: Clone the Merge-Simulation Remote on main Regardless of init.defaultBranch On CI the bare remote's HEAD names master, so the hosting clone had an empty HEAD and `git merge --squash` refused to run. --- packages/code/src/worktree-retirement.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/code/src/worktree-retirement.test.ts b/packages/code/src/worktree-retirement.test.ts index 4c81b2e5..f10ed8f3 100644 --- a/packages/code/src/worktree-retirement.test.ts +++ b/packages/code/src/worktree-retirement.test.ts @@ -258,7 +258,7 @@ test('a squash- or rebase-merged branch whose remote branch was deleted is retir await git(ahead, 'commit', '-q', '-m', 'later, also landed on main but never pushed here'); const hosting = join(parent, 'hosting'); - await git(parent, 'clone', '-q', join(parent, 'remote.git'), hosting); + await git(parent, 'clone', '-q', '-b', 'main', join(parent, 'remote.git'), hosting); await git(hosting, 'merge', '-q', '--squash', 'origin/squashed'); await git(hosting, 'commit', '-q', '-m', 'squash merge'); await git(hosting, 'cherry-pick', 'origin/rebased');