bug: long-running processes lose provider access after credential refresh

[mrunalp]

Agent Diagnostic

• Skills used: openshell-cli for the end-to-end sandbox/provider investigation and create-github-issue for this report.
• OpenShell version tested: locally built development binaries reporting openshell 0.0.103-dev.6+gc825b1f8; the source-level proof of concept was validated at checkout 8b7c444c (v0.0.103-17-g8b7c444c) with local uncommitted changes.
• Latest release checked: v0.0.106 (published 2026-08-14). Its release notes do not list a fix for long-running process placeholders. The release artifact was not retested end to end because the reproduction uses a locally built Podman gateway, custom provider refresh configuration, and a local proof-of-concept build.
• Known fixes/designs reviewed: feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306, PR feat(providers): support sandbox provider attach lifecycle #1242, and PR feat(proxy): bind static credentials to provider endpoints #2510. feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306 owns gateway-side token refresh; its long-running process report describes the same remaining failure after refresh succeeds.
• Possible duplicates reviewed: Enhanced Provider Management #896, Support Codex OAuth in OpenShell using provider-backed auth.json placeholders #988, Provider-managed OAuth credential refresh #1066, feat: support provider attach and detach for running sandboxes #1171, feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, and feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306. No dedicated issue was found for the expired retained placeholder generation or for an identity-stable, revocable workload handle.
• Findings: the gateway refreshed the OAuth access token successfully; a fresh sandbox exec inherited the new provider-environment placeholder and completed a metadata-only request; the existing long-running process retained the old placeholder and failed locally with credential_unavailable after its access token expired. The request did not reach Google. Restarting only the long-running process fixed the request.
• Code diagnosis: provider snapshots are revision-scoped and retain eight generations. Exact lookup of a retained old placeholder wins before same-identity fallback, then fails closed when that generation's token is expired. Routine refresh therefore changes the handle seen by future processes but cannot change the environment inherited by an existing process.
• Proof of concept: keeping an endpoint-bound handle stable across same-identity token updates passed focused tests for an expired initial token, 12 refreshes, non-monotonic revisions, endpoint rebinding, per-key identity replacement, detach/recovery, and replacement revocation. The focused suite passed 35 tests; the full openshell-core suite passed 392 tests with 6 ignored; clippy and relevant crate checks passed.
• Remaining reason for filing: the POC demonstrates the resolver behavior but is not production-ready. The current gateway identity is only <provider object id>:<credential key>, and the POC chooses its stable handle in supervisor memory. Production needs a complete identity-epoch contract and a durable or authenticated handle that survives supervisor/sidecar restart without weakening revocation or endpoint binding.

Description

Actual behavior: A long-running process can lose provider access after OpenShell successfully refreshes a short-lived access token. The process retains a placeholder such as openshell:resolve:env:v1_GOG_ACCESS_TOKEN. A later refresh installs revision v2 for new process launches, but the existing process and its children continue using v1. Once the access token stored in retained generation v1 expires, exact placeholder resolution returns unavailable even though the current same-identity generation contains a valid replacement token.

This affects any process that captures provider environment once and lives across token rotations: agent gateways, editor servers, open PTYs, terminal multiplexers, language servers, and background workers. The application may surface the local proxy failure as an HTTP 500, making it look like an upstream provider outage.

Expected behavior: Routine same-identity access-token refresh should update the credential value behind an identity-bound workload handle. Existing authorized processes should continue working without receiving raw tokens, refresh material, or environment mutation. The handle must fail closed on expiry when refresh fails and must be revoked when its authorization identity or boundary changes.

Root cause

The current model couples two lifetimes that need different semantics:

Layer	Required lifetime
OAuth refresh material	Gateway-only grant lifetime
OAuth access token	Short token generation, often about one hour
Workload placeholder	Provider credential identity epoch

ProviderCredentialState retains at most eight revision-scoped credential generations. Exact placeholder lookup takes precedence over fallback. If the exact generation still exists but its secret has expired, resolution fails. Same-identity fallback can occur only after an old generation is evicted, so a long-running process can fail for several refresh cycles and later recover as an accidental consequence of retention.

Proposed remediation

Issue one opaque, revocable handle per sandbox/provider credential identity epoch and keep it stable across routine access-token refreshes. Refresh should atomically replace the current access token and expiry behind that handle. Provider environment revision can continue changing for polling, cache invalidation, non-secret environment changes, and future launches; it should not define the lifetime of an already-issued workload authorization handle.

The identity epoch should bind to the equivalent of:

• sandbox identity;
• provider instance and credential key;
• verified OAuth subject/account/tenant identity where available;
• OAuth client and scope or authorization-boundary digest;
• endpoint-binding identity;
• an explicit revocable epoch identifier.

The gateway should issue a durable opaque handle or enough authenticated identity-epoch metadata to reconstruct it consistently. Choosing the stable revision only in supervisor memory is insufficient because a supervisor or sidecar may restart while the workload survives.

Routine same-identity access-token rotation updates the value behind the existing handle. Detach, provider deletion/replacement, account or tenant change, material client/scope change, endpoint-binding identity change, explicit revocation, or sandbox deletion rotates or revokes the epoch. A refresh failure may preserve the current access token only until its recorded expiry, then resolution must fail closed.

This design must preserve the security invariant established by PR #2510: a stale process holding a handle for one provider identity must not resolve a replacement provider's credential merely because both use the same environment key.

Alternatives considered

• Allow fallback from an expired exact generation to the current token only when identity matches. This is a smaller change, but the current identity (provider object id plus credential key) does not include verified subject, tenant, client, scopes, or a binding digest, so reauthorization can be misclassified as continuity.
• Launch every provider client through a fresh supervisor exec. This works around stale environments but requires application-specific brokering and does not fix editors, shells, or ordinary child-process inheritance.
• Restart long-running processes after every refresh. This is the current workaround and interrupts sessions, sockets, editor state, and in-flight work roughly once per access-token lifetime.
• Keep old access tokens usable indefinitely. Rejected: expiry must remain fail-closed, and refresh material must never enter the workload.

Reproduction Steps

1. Configure a provider with gateway-owned OAuth refresh material and a short-lived injectable access-token credential.
2. Attach it to a running sandbox and start a long-running parent process from a supervisor-created exec.
3. Confirm, without printing credential values, that the process inherited revision-scoped placeholder P1.
4. From that process, run a metadata-only provider operation and confirm success.
5. Wait for a successful gateway refresh and provider-environment revision change.
6. Confirm that a fresh supervisor exec inherits placeholder P2 and succeeds with the same metadata-only operation.
7. After the token associated with P1 expires, launch the provider client from the original parent process.
8. Observe local credential_unavailable/expired-placeholder diagnostics and confirm the request does not reach the provider.
9. Restart only the long-running parent through a fresh supervisor exec and observe that the operation succeeds.

For Google Workspace with gog, a content-free probe is:

gog --readonly --no-input --json gmail labels list >/dev/null

Do not print environment values or token-bearing request data while reproducing.

Environment

• OS: Fedora Linux 44 (Workstation Edition)
• Sandbox backend: local Podman
• OpenShell end-to-end binary: 0.0.103-dev.6+gc825b1f8
• POC source checkout: 8b7c444c (v0.0.103-17-g8b7c444c) with local uncommitted POC changes
• Workload: OpenClaw launching gog for a metadata-only Google Workspace Gmail request
• Credential lifecycle: gateway-owned OAuth refresh token; short-lived access token exposed to the sandbox only through an OpenShell placeholder
• Latest release checked: v0.0.106; release notes reviewed, no matching fix listed; release artifact not retested for the reason given in Agent Diagnostic
• Possible duplicates checked: yes; Enhanced Provider Management #896, Support Codex OAuth in OpenShell using provider-backed auth.json placeholders #988, Provider-managed OAuth credential refresh #1066, feat: support provider attach and detach for running sandboxes #1171, feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, and feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306

Logs

# Redacted OpenShell diagnostics; no credential values or mailbox data.
credential resolution rejected: credential is expired location=resolve_placeholder
reason=credential_unavailable

# Control observations:
gateway refresh status: successful
fresh supervisor exec using current placeholder: request successful
existing long-running process using retained placeholder: request failed locally
provider request observed: no
restart of long-running process only: request successful

Acceptance Criteria

• A process started before refresh continues making authorized requests through at least 12 consecutive same-identity access-token rotations.
• Access tokens and refresh material never enter the process environment; refresh tokens, client secrets, and service-account private material remain non-injectable gateway state.
• Handles survive supervisor/sidecar restart when the workload survives.
• Detach, deletion, explicit revocation, or provider/account/subject/tenant/authorization replacement invalidates old handles.
• An old handle cannot cross provider identity, OAuth subject, client/scope authorization boundary, or endpoint binding.
• Failed refresh preserves the current token only until recorded expiry, then fails closed without a failure window tied to generation eviction.
• Endpoint and L7 policy checks still run before substitution; endpoint-bound credentials cannot be replayed to another host, port, or path.
• Docker, Podman, Kubernetes sidecar, and VM supervisor topologies have equivalent behavior.
• E2E coverage uses a fake short-lived token issuer and includes long-running-process, refresh, revocation, replacement, refresh failure, and supervisor/sidecar restart cases.
• OCSF/application diagnostics distinguish local credential unavailability from an upstream HTTP 500 without logging placeholders, tokens, keys, query data, or user content.

Related Work

• feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306 owns gateway-side refresh material and token rotation.
• Matching long-running-process report on #1306
• feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264 proposed stable request-time resolver updates.
• PR feat(providers): support sandbox provider attach lifecycle #1242 introduced provider attach/detach generation snapshots.
• PR feat(proxy): bind static credentials to provider endpoints #2510 added provider identity and endpoint binding that this fix must preserve.

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue.
• I loaded relevant skills.
• I checked the latest OpenShell release and explained why the artifact was not retested.
• I searched existing issues for possible duplicates.
• The diagnostic explains why the issue remains unresolved.

[mrunalp]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: High
Confidence: Medium — the resolver path is clear, but handle representation, compatibility, and sidecar-restart coverage require careful implementation.

Summary

Implement Option A only for gateway-refresh-managed, endpoint-bound credentials. Persist an explicit authorization epoch, derive a stable opaque workload handle from that epoch plus the sandbox/provider/key/endpoint authorization boundary, and update the current access token behind the handle during routine refresh. Unmanaged credentials remain revision-scoped.

Scope

• proto/openshell.proto: add the persisted refresh authorization epoch and gateway-issued stable-handle metadata, with compatibility and revocation semantics.
• crates/openshell-server/src/provider_refresh.rs: create a new epoch on every explicit refresh configuration, preserve it through automatic/manual rotation, and handle legacy empty epochs safely.
• crates/openshell-server/src/grpc/provider.rs: map primary and additional refresh outputs to their authorization epoch and derive canonical boundary-bound handles without credential material.
• crates/openshell-server/src/grpc/policy.rs: include refresh epoch/configuration state in provider-environment revision computation and pass sandbox identity into handle derivation.
• crates/openshell-core/src/provider_credentials.rs: install gateway-issued handles, update values behind stable handles, and revoke old handles on identity/boundary changes while preserving legacy behavior.
• crates/openshell-core/src/secrets.rs: resolve opaque stable handles with current expiry and endpoint-scoping semantics; do not retain old stable values as generations.
• e2e/: add a fake short-lived token/resource flow for a long-running parent process and refresh/revocation behavior where feasible in the existing harness.
• architecture/sandbox.md and docs/sandboxes/providers-v2.mdx: document handle lifetime, rollout compatibility, refresh, revocation, and restart behavior.

Implementation Steps

1. Extend refresh state with a persisted authorization epoch. Generate a new epoch on every explicit ConfigureProviderRefresh; preserve it during automatic and manual access-token rotation. Associate co-minted additional outputs with the same epoch. For pre-upgrade records with an empty epoch, use a fail-safe migration path without exposing material.
2. Load refresh identity metadata during provider-environment assembly. Canonically sort/normalize endpoint bindings and derive a domain-separated opaque handle from sandbox ID, provider object ID, credential key, authorization epoch, and endpoint-binding digest.
3. Include refresh-state epoch/configuration generation in provider_env_revision, ensuring explicit reconfiguration, refresh deletion, and authorization-boundary changes reach running supervisors before another token mint.
4. Replace the POC's supervisor-selected first revision with gateway-issued handles. Stable handles exist only for refresh-managed endpoint-bound credentials; refresh atomically replaces their value/expiry, revoked handles fail closed, and legacy/unmanaged placeholders retain revision-scoped behavior.
5. Preserve the Kubernetes process/network split: the process-side control channel continues carrying only prepared child environment values, and a restarted network sidecar reconstructs the same handle from gateway state.
6. Preserve typed endpoint-mismatch versus credential-unavailable diagnostics across request paths without logging handles, keys, tokens, or user data.
7. Update architecture and user documentation, including the one-time restart requirement for processes that began with legacy revision placeholders.

Test Plan

• Unit tests: configuration creates/rotates epochs; worker/manual refresh preserves them; additional outputs share an epoch; legacy records migrate safely; canonical handle derivation is ordering-independent and rotates for sandbox/provider/key/epoch/endpoint changes; 12 token rotations keep one handle; expired tokens fail closed; replacement/detach/reconfigure revoke; unmanaged placeholders remain revision-scoped.
• Integration tests: compare GetSandboxProviderEnvironment before/after refresh and reconfiguration; rebuild ProviderCredentialState from a later response and resolve the original handle to simulate network-sidecar restart; cover refresh deletion, endpoint replacement, inline credentials, and credential-driver-backed access tokens.
• E2E tests: use a fake short-lived issuer and resource endpoint with a long-running parent that retains its original environment, verifying current-token substitution across repeated refreshes plus failure/expiry, reconfigure, detach, replacement, and supported driver/topology lanes. Never print tokens, handles, or response content.
• Local acceptance: rebuild the Podman gateway/supervisor, restart OpenClaw once to acquire the new handle, rotate the Google credential repeatedly, and run gog --readonly --no-input --json gmail labels list >/dev/null from the same OpenClaw process after the original token expires. No OpenClaw restart should be required for routine refresh; explicit reconfiguration should revoke the old process handle.

Risks & Open Questions

• Use a full opaque handle rather than reducing the boundary to a 64-bit revision.
• Existing processes with legacy placeholders cannot become restart-durable retroactively and require one restart during rollout.
• A verified OAuth subject is not currently stored. Rotating an unguessable persisted epoch on every explicit configure safely treats reauthorization as a new identity.
• Refresh-state and provider revision updates must be ordered fail-securely: partial failure may revoke unnecessarily but must not let an old handle resolve a new authorization.
• Endpoint canonicalization must be deterministic across HA gateway replicas.
• Explicit refresh deletion revokes the stable handle immediately; an unchanged reattachment reauthorizes the sandbox unless a future attachment-epoch design is required.

Documentation Impact

Update architecture/sandbox.md and docs/sandboxes/providers-v2.mdx. No gateway TOML, driver option, Helm value, or default changes are planned, so docs/reference/gateway-config.mdx is unaffected.

LSM Compatibility

No SELinux or AppArmor behavior change is expected. This does not change process identity, /proc access, binary execution, namespaces, or inter-process visibility.

---

Revision 1 — initial Option A production plan

[mrunalp]

Implementation is ready for review in #2780.

Validation completed:

• Added a Podman E2E that keeps one workload process alive across 12 access-token rotations, verifies it resolves the latest token, reconstructs supervisor credential state, and confirms explicit refresh reconfiguration revokes the old handle.
• Verified the same behavior with the original OpenClaw + gog scenario: one already-running OpenClaw gateway accessed Gmail before and after an OpenShell provider refresh without restarting. The check discarded command output and captured no mailbox content.
• mise run pre-commit, mise run test, and mise run ci all pass locally.

Commit: a505e8d9eb92235e60bdb5dc421ade54836cd6ed