feat: store provider refresh credentials in credential drivers

[mrunalp]

User Story

As an OpenShell gateway operator using an external credential store, I want provider refresh credentials to use the configured credential driver, so that long-lived OAuth secrets follow the same storage, access-control, audit, and lifecycle boundary as other provider credentials.

Problem Statement

Provider refresh configuration currently keeps secret refresh material, including OAuth client secrets and refresh tokens, in the gateway refresh-state record. Configuring Vault or another credential driver for provider credentials does not move that long-lived refresh material into the driver.

The gateway should retain only non-secret refresh metadata, scheduling state, and opaque credential handles. It should resolve refresh inputs through the configured credential driver when minting a new access token.

Impact / Why This Matters

Operators adopt Vault or another external credential driver to centralize secret access controls, audit trails, rotation, and deletion. Keeping refresh tokens in gateway state creates a second secret-storage boundary and means database backups still carry long-lived refresh material even when Vault is configured.

The current workaround is to rely on gateway database encryption or manage token refresh outside OpenShell. Database encryption does not provide the configured external store's lifecycle and audit properties. External refresh also gives the sandbox or another process more credential authority and duplicates OpenShell's refresh scheduler.

Proposed Design

When an operator configures a refreshable provider:

• OpenShell stores secret refresh inputs through the active credential driver and retains only opaque handles in gateway refresh state.
• Non-secret inputs, such as an OAuth client ID, may remain in refresh metadata.
• Scheduled and forced refreshes resolve long-lived inputs from the credential driver, call the provider token endpoint, and store the minted access token through the credential driver.
• If the provider returns a rotated refresh token, OpenShell replaces the stored refresh token safely and removes the superseded secret. If no replacement is returned, the existing refresh token remains valid.
• Reconfiguration, deletion, and failed or superseded rotations do not leak orphaned secret objects or replace a last-known-good configuration with partial state.
• Existing inline refresh configurations migrate to the active credential driver without requiring users to reauthorize. Legacy driver handles remain resolvable during a deliberate driver transition.
• API and CLI status operations expose handles and operational state, never secret values.
• Sandboxes continue to receive only short-lived credential substitution at an authorized request boundary; they never receive refresh tokens.

The existing provider refresh configure, rotate, status, and delete workflow should remain stable.

Acceptance Criteria

• Configuring OAuth refresh material with Vault active stores the client secret and refresh token in Vault and stores only handles in gateway refresh state.
• Scheduled and forced refreshes resolve refresh inputs from Vault and store the minted access token through the active credential driver.
• A refresh-token replacement returned by the OAuth provider atomically supersedes the previous Vault object; a response without a replacement preserves the existing token.
• Failed, deleted, or concurrently superseded refresh attempts preserve the last-known-good state and clean up staged credentials.
• Reconfiguring or deleting refresh state removes owned credential objects, with retryable cleanup for transient driver failures.
• Existing inline refresh secrets migrate to the active credential driver without reauthorization and without exposing values through APIs or logs.
• Credential-driver transitions retain access to legacy handles until migration and cleanup finish.
• Unit tests cover storage, resolution, migration, rotation, concurrency, and cleanup behavior.
• An end-to-end test demonstrates a Vault-backed OAuth refresh, access-token rotation, and successful sandbox request before and after rotation.
• Architecture, gateway configuration, provider workflow, and local Vault validation documentation describe the behavior and operational caveats.

Alternatives Considered

• Keep refresh material encrypted in the gateway database. This preserves a separate secret boundary and does not provide Vault's policy, audit, and lifecycle controls.
• Store only the refresh token in Vault. Leaving the OAuth client secret or minted access token elsewhere creates inconsistent ownership and cleanup behavior.
• Let gog or another sandbox tool refresh locally. This requires giving the sandbox a refresh token and undermines the gateway-managed credential boundary.
• Configure a separate refresh-secret backend. Reusing the credential-driver contract avoids a second driver model and gives provider credentials one storage policy.

Agent Investigation

This builds on PR #2780, which keeps refresh credential handles stable across access-token rotation.

The implementation plan is captured in this design gist.

A local validation used a loopback-only Vault 1.20.4 dev container, an isolated Podman gateway, the gog Gmail read-only provider, and OpenClaw. It verified that:

• existing refresh inputs migrated into Vault;
• forced rotation succeeded and did not accumulate stale access-token objects;
• a running gog sandbox could access Gmail before and after rotation without receiving the refresh token; and
• OpenClaw completed the constrained Gmail probe through the same Vault-backed provider.

The migration exercise also identified that refresh-state list paths must hydrate the authoritative database resource version before compare-and-swap updates.