feat(cli): add sandbox prune command to delete ERROR-phase sandboxes

[engelmi]

Problem Statement

When sandboxes fail during provisioning or runtime, they remain in the ERROR phase indefinitely. Users currently have two options:

• delete them individually by name with sandbox delete <name>, or
• wipe everything with sandbox delete --all

There is no way to clean up only the broken sandboxes while leaving healthy ones running.

This is especially painful in environments where multiple sandboxes are running concurrently - a single failed experiment shouldn't require manually identifying and deleting each errored sandbox, and --all is too destructive.

Proposed Design

Add openshell sandbox prune - a CLI command backed by a new PruneSandboxes gRPC RPC that performs server-side filtering and deletion. The semantics are similar to podman container prune and docker container prune.

Proto:

• New PruneSandboxes RPC on the OpenShell service with sandbox:write scope and user workspace role (matching DeleteSandbox).
• PruneSandboxesRequest takes workspace and all_workspaces (admin-only, mirrors ListSandboxes).
• PruneSandboxesResponse returns repeated string pruned_names and repeated string failed_names for best-effort semantics - one sandbox failing to delete doesn't abort the rest.

Alternatives Considered

1. Client-side filtering - CLI lists all sandboxes, filters for ERROR, then issues individual DeleteSandbox RPCs. Simpler (no proto changes), but introduces a TOCTOU race between list and delete, produces N round trips, and pushes logic into every client (CLI, SDK, Python SDK).
2. Add a phase filter to ListSandboxes. More general, but still requires client-side delete loop. The prune operation is common enough to warrant a dedicated RPC that does both steps atomically.
3. Extend DeleteSandbox with a phase filter. Overloads the existing RPC's semantics. A separate RPC is cleaner and doesn't risk breaking existing callers.

Agent Investigation

• Explored the full DeleteSandbox call chain: CLI (run::sandbox_delete) → gRPC
  → gateway handler (handle_delete_sandbox) → ComputeRuntime::delete_sandbox()
  → compute driver. The handler is a thin authz wrapper; all complexity lives in
  the compute layer which is safe to reuse.
• Confirmed SandboxPhase enum has ERROR = 3 and the established pattern for
  checking phase is SandboxPhase::try_from(sandbox.phase()).ok() ==
  Some(SandboxPhase::Error).
• Confirmed the gateway's existing reconciliation loop
  (reconcile_store_with_backend) already prunes orphan sandboxes on a timer, but
  does not prune by phase — this is a separate concern.
• Confirmed ListSandboxesRequest has no phase filter field, so client-side
  filtering would require fetching all sandboxes.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[engelmi]

Implementation in: https://github.com/engelmi/OpenShell/tree/add-sandbox-prune-command

[politerealism]

Hey Michael — took a proper look at add-sandbox-prune-command, both a manual pass and a more thorough automated review. Good news: the overall approach is sound (reusing state.compute.delete_sandbox instead of reimplementing cleanup was the right call). A few things worth fixing before this goes up as a real PR, roughly in priority order:

Two real bugs:

1. --all-workspaces doesn't work as shipped — the CLI sends the default workspace name alongside all_workspaces: true, and the server rejects that exact combination as mutually exclusive. Looks like it was only tested by constructing server requests directly, never through the actual CLI flag.
2. The SDK's top-level prune_sandboxes() will hit permission_denied for any non-admin caller — it sends an empty workspace expecting server-side default resolution, but authorization runs on the raw empty string before that normalization happens. This is masked because all the added tests use a helper that grants platform-admin to every principal.

Worth addressing before merge:
3. The sandbox listing uses a single fixed limit=1000, offset=0 call with no pagination — handle_list_sandboxes in the same file already has the cursor-based pattern for this, worth reusing so large workspaces don't silently drop sandboxes past the first page.
4. The delete loop processes sandboxes sequentially. delete_sandbox already does its own per-sandbox locking, so nothing requires serializing the batch — bounded concurrency would avoid multiplying driver round-trip latency and repeated lock contention for a big prune.
5. Docs — docs/sandboxes/manage-sandboxes.mdx already has a Delete/Lifecycle section this should extend.

Minor:
6. Under --all-workspaces, the last-sandbox cache gets cleared using the CLI's configured workspace rather than each sandbox's actual origin workspace — moot until #1's fixed, but will resurface once it is.
7. There's an unrelated mise.lock diff (an sccache block) worth dropping.

One more thing, not a fix, just context: I've been working with Drew on the RFC 0012 isolation-backend rollout, and there's an open question relevant to this design — does the reconciliation loop detect a half-created supervisor/workload pair under the new topology, and is ERROR phase well-defined for that case? Still waiting on his answer, but wanted you to know it's in flight since your prune-by-ERROR-phase design leans on that being settled.

Happy to help work through any of these if useful — this is close.

@sjenning since you're a maintainer here — could you take a look and weigh in on whether this overall design (a dedicated PruneSandboxes RPC, best-effort per-sandbox deletion, ERROR-phase filtering) is something you'd be open to accepting once the issues above are addressed?

[sjenning]

I think I can say that we don't want to implement a new gRPC interface for this.  Especially when it can be done client side with something like
for i in $(openshell sandbox list | grep -i error | cut -f1 -d' '); openshell sandbox delete $i; done

At the very most, we could implement this in the CLI/SDK as a client helper

It seems this was covered and discounted in the alternatives

Alternatives Considered
Client-side filtering - CLI lists all sandboxes, filters for ERROR, then issues individual DeleteSandbox RPCs. Simpler (no proto changes), but introduces a TOCTOU race between list and delete, produces N round trips, and pushes logic into every client (CLI, SDK, Python SDK).

But I think this is the way. openshell sandbox delete --all already works this way. Maybe a openshell sandbox delete --prune where it is a phase filtered version of --all.

[politerealism]

I verified this as well. I'd suggest pivoting this to a CLI/SDK only sandbox prune command which would list with a phase filter or filter client side after listing, same as --all does today, then loop through the existing delete_sandbox path. Should be a much smaller diff than the RPC based version, and inherits correctness from code that's already there. @engelmi

[engelmi]

Thanks for the feedback! @politerealism @sjenning
I'll prepare a PR for with the discussed --prune option to the delete command on the client-side CLI.

[engelmi]

@politerealism @sjenning Opened PR #3378 to implement the --prune flag. I diverged a bit from the discussed filtering by not only deleting sandboxes in the ERROR phase, but also in other "inactive" phases such as Completed. Please have a look and let me know what you think.