fix(workspaces): wait for the Docker bootstrap before using a woken workspace

[Lob26]

The gap

DockerWorkspaceRuntime.create proved a container ready before handing it back. wake only started it:

if (!existing.State?.Running) await this.docker.getContainer(existing.Id).start();
return this.handle(workspace, existing.Id, names);   // no readiness probe

But the bootstrap deletes its own readiness marker on every start, and only then brings up dockerd, chowns /workspace, fixes the socket permissions and launches the preview gateways:

rm -f /workspace/.facility/runtime-ready
chown -R node:node /workspace
rm -f /var/run/docker.sock
dockerd ... &
until docker info; do ...; done
... facility-preview-gateway ...
touch /workspace/.facility/runtime-ready

So docker start returning says nothing about whether the workspace can be used, and both callers of wake act on the handle immediately:

• exec → wake → run the agent's command. This is where docker, docker compose and a half-chowned /workspace fail intermittently after a resume, a host restart, or an OOM restart — the socket has just been deleted and not yet recreated.
• preview.open → wake → read the published ports. Docker binds the host port at container start, so the endpoint URL is valid while the gateway process behind it is not yet listening.

VercelWorkspaceRuntime.wake already re-runs initializeAndHandle, so the two providers were not honouring the same contract — and Docker is the default for self-hosting and docker-compose.

This is the level-triggered version of the same bug shape as #319/#320: wake was reacting to the edge ("start returned") instead of observing the state ("the marker is there and dockerd answers").

The change

wake proves readiness too, through a shared ensureReady.

Readiness only ever moves forward within one container start, so each start is proven once and remembered by container name → ${Id}:${StartedAt}:

• a resumed or replaced container gets a new StartedAt (or a new Id) and is proven again;
• exec — which calls wake per command — does not pay for a probe per command, only the inspect it already needed;
• a container that someone else started and that is still booting is also waited for, which the narrower "only wait if we started it" fix would miss. preview.open and a turn's exec genuinely race here.

The map is keyed by the workspace's stable container name and cleared in destroy, so it is bounded by live workspaces.

ensureReady also fails loudly when the bootstrap exited instead of becoming ready, rather than returning a handle to a dead container.

Test

New services/api/test/workspace-docker.test.ts, using an injected fake dockerode — the same shape as the existing workspace-vercel.test.ts — so it runs in the default suite with no Docker daemon:

• a stopped container is started and not returned until the readiness probe exits 0;
• a container someone else started and that is still booting is waited for, without a second start;
• three wake calls against the same container start probe once;
• a bootstrap that exits raises workspace_initialize_failed instead of returning a handle.

All four fail on unmodified main (git stash-ed the source file to check) and pass with the change.

Verification

Run on Windows 11, Node 26, against the compose Postgres.

• pnpm --filter @facility/api typecheck — clean
• pnpm lint — clean
• pnpm guards — 2 guards, 0 failed
• pnpm check:unused — clean
• vitest run test/workspace-docker.test.ts test/workspace-runtime.test.ts test/workspace-preview.integration.test.ts test/story-workspaces.integration.test.ts — 21 passed
• Full services/api suite: 258 passed, 14 failed — every failure also fails on unmodified main in this environment (the POSIX-shell fakes in workspace-vercel-bootstrap, agent-engines, facility-012.e2e, project-environment, turn-dispatcher, i.e. the #227/#241 Windows family), plus one load-dependent flake in workspace-runtime.test.ts that passes 3/3 in isolation and only exercises FakeWorkspaceRuntime.

What I could not verify here: the real Docker path. FACILITY_E2E_DOCKER=1 pnpm test:e2e-workspace needs a Linux daemon that can run the privileged runner image with nested dockerd, and this machine has rootless Podman. The probe itself is the one create has always used, so the risk is in ensureReady's call sites rather than in the probe, but a real suspend → wake → exec on CI is the check that would actually close it.

Claude Code helped

[adrian-lorenzo]

Thanks for addressing this.

A real restart still fails because /var/run/docker.pid survives the stop and blocks dockerd from starting. Please handle stale PID cleanup and add a real suspend/wake regression test so the resume fix works end to end.

[Lob26]

Addressed at 14ef273. You were right, and I confirmed the mechanism on a real container rather than assuming it: a value written to /var/run/docker.pid is still there after a stop and start, so dockerd refuses to boot and the bootstrap dies waiting. It now clears that state, containerd's exec root included, before starting the daemon. The end-to-end case suspends and wakes the same container twice, since the first wake writes a pidfile of its own. I could not build the runner image locally, so CI is the authority on that half. Thank you.