Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/test-release-canary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--namespace openshell --create-namespace \
--set server.disableTls=true \
--set server.telemetryEnabled=false \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

kubectl wait --namespace openshell \
Expand Down
33 changes: 24 additions & 9 deletions .github/workflows/release-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,28 @@ jobs:
launchctl setenv OPENSHELL_COMPUTE_DRIVER vm
launchctl setenv OPENSHELL_TELEMETRY_ENABLED "$OPENSHELL_TELEMETRY_ENABLED"

- name: Install and check status
- name: Install and check gateway status
run: |
set -euo pipefail
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
openshell --version
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"

# GitHub-hosted macOS runners do not expose the Hypervisor.framework
# support libkrun needs. Sandbox launch is covered by the VM E2E lane.
- name: Collect Homebrew diagnostics
if: failure()
run: |
set +e
brew services info openshell
for log in \
"$(brew --prefix)/var/log/openshell/openshell-gateway.out.log" \
"$(brew --prefix)/var/log/openshell/openshell-gateway.err.log"; do
if [ -f "$log" ]; then
echo "--- $log ---"
tail -n 300 "$log"
fi
done

ubuntu:
name: Ubuntu Docker
Expand All @@ -68,7 +81,7 @@ jobs:
set -euo pipefail
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
sandbox="rc-${GITHUB_RUN_ID}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
Expand Down Expand Up @@ -153,7 +166,7 @@ jobs:
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
OPENSHELL_VERSION="$OPENSHELL_VERSION" \
OPENSHELL_TELEMETRY_ENABLED="$OPENSHELL_TELEMETRY_ENABLED" \
CANARY_SANDBOX="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
CANARY_SANDBOX="rc-${GITHUB_RUN_ID}" \
INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \
bash -s <<'EOF'
set -euo pipefail
Expand Down Expand Up @@ -235,7 +248,7 @@ jobs:
- name: Create and exercise a sandbox
run: |
set -euo pipefail
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
sandbox="rc-${GITHUB_RUN_ID}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
Expand Down Expand Up @@ -292,7 +305,9 @@ jobs:
--version 0.0.0-dev \
--namespace "$RELEASE_NAMESPACE" --create-namespace \
--set server.disableTls=true \
--set server.auth.allowUnauthenticatedUsers=true \
--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

- name: Verify gateway pod is Ready
Expand Down Expand Up @@ -333,7 +348,7 @@ jobs:
set -euo pipefail
openshell gateway add http://127.0.0.1:8080 --local --name "$KIND_GATEWAY_NAME"
openshell status
sandbox="release-canary-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
sandbox="rc-${GITHUB_RUN_ID}"
openshell sandbox create --name "$sandbox" --detach
openshell sandbox exec --name "$sandbox" --no-tty -- true
openshell sandbox delete "$sandbox"
Expand Down
5 changes: 3 additions & 2 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ creation. Drivers that enforce policy outside the standard supervisor fetch
later revisions through `GetSandboxConfig` and acknowledge them through
`ReportPolicyStatus`.
Process-identity omissions are preserved across this boundary so every driver
can apply its native image or runtime defaults. Driver-requested listeners are
structurally validated and remain restricted to sandbox callback RPCs.
can apply its native image or runtime defaults. Drivers connect supervisors to
the operator-configured gateway endpoint; they do not request additional
gateway listeners.

Canonical main-process support is part of the `ComputeDriver` contract. Every
in-tree and extension driver must forward the exact specification; it is not an
Expand Down
61 changes: 22 additions & 39 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ attachments; and asks compute runtimes to create or delete sandbox workloads.

## Responsibilities

- Authenticate clients and sandbox callbacks.
- Authenticate clients and sandbox supervisor sessions.
- Serve gRPC APIs for sandbox lifecycle, provider management, policy updates,
settings, logs, watch streams, and relay forwarding.
- Serve HTTP endpoints for health, WebSocket tunnels, and edge-auth flows.
Expand Down Expand Up @@ -49,7 +49,7 @@ versions fail before runtime construction, and driver settings belong only to
Package lifecycle code may replace an exact package-generated v1 default, but
it preserves edited configurations for explicit operator migration.

Gateway listener TLS and sandbox callback TLS are separate inputs. A selected
Gateway listener TLS and sandbox supervisor TLS are separate inputs. A selected
local Docker, Podman, or VM driver requires a complete guest bundle whenever
the gateway listener uses TLS; package-managed local TLS can supply that bundle.
Kubernetes instead projects guest credentials through its configured Secret.
Expand Down Expand Up @@ -122,24 +122,16 @@ operations require it. The gateway authorizes the selected scope before
performing resource lookup so malformed, unsupported, and unauthorized scopes
have consistent behavior across resource types.

Docker and Podman report the local address through which their sandboxes can
reach the gateway. When the primary listener covers that address, the gateway
reuses it; sandbox JWT authentication and its RPC allowlist remain the callback
authorization boundary. When the primary listener does not cover the address,
the gateway adds a callback-only listener. Additional callback listeners accept
only gRPC methods classified as sandbox-callable by the gateway's generated
authorization metadata. They reject user and administrator APIs, health,
reflection, and HTTP routes before normal request
authentication. The operator-configured primary listener retains the full
multiplexed API surface.

The `rpc_auth` classification is also the source of truth for negotiated
listener exposure: marking an RPC as `sandbox` or `dual` makes it callable on
these listeners. Review such changes as both authorization and network-surface
changes. Listener requirements are currently authorized only for the built-in
Docker and Podman drivers. Operator-granted listener capabilities for external
drivers are tracked in
[#2539](https://github.com/NVIDIA/OpenShell/issues/2539).
Docker and Podman supervisors use host networking and connect through the
gateway's primary listener. On Linux, local supervisors use the primary
loopback endpoint. Sandbox JWT authentication and the generated sandbox RPC
allowlist remain the authorization boundary; the gateway does not negotiate or
bind compute-driver-specific listeners.

The `rpc_auth` classification is the source of truth for supervisor access.
Marking an RPC as `sandbox` or `dual` makes it callable by an authenticated
sandbox principal on the primary listener. Review such changes as
authorization-surface changes.

Operators can configure a gateway-wide gRPC request rate limit. The limit is
applied only to gRPC API traffic after protocol multiplexing; health, metrics,
Expand Down Expand Up @@ -1014,25 +1006,16 @@ system entry instead of pretending to delete package-manager owned state.

- Gateway TLS and client certificate distribution are deployment concerns owned
by the operator or packaging layer.
- Compute runtimes own the mechanics of starting workloads and injecting
callback configuration. Local Docker, Podman, and VM callback endpoints can
be derived from their fixed host aliases. Kubernetes requires an explicit
endpoint from driver placement; Helm renders it from the gateway Service
name and namespace rather than inferring it from sandbox placement.
- Docker-backed local gateways use Docker's `host-gateway` callback alias on
macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
reachability and add a listener only when the primary does not cover it.
Native Linux Docker may expose an additional bridge-gateway listener because
the host can bind that bridge IP.
- Podman-backed macOS gateways use gvproxy's host-loopback IP for sandbox host
aliases by default so stale Podman machine images do not need Podman's
`host-gateway` resolver. Linux Podman keeps the resolver unless
`host_gateway_ip` is configured. Rootful Podman can request its exact bridge
gateway listener. Rootless Podman explicitly reporting pasta requests the
private IPv4 source selected by the host default route rather than an
arbitrary private interface. Slirp4netns, other helpers, and missing helper
metadata fail closed for local callbacks until a rootless-network namespace
relay is available.
- Compute runtimes own the mechanics of starting workloads and injecting the
gateway endpoint. Docker and Podman supervisors use host networking; local
Linux supervisors use the gateway's primary loopback endpoint. Kubernetes
uses the gateway Service rendered by Helm. VM supervisors use their
runtime-specific host route.
- Docker Desktop requires host networking to be enabled and cannot combine it
with Enhanced Container Isolation. Set an explicit remote `grpc_endpoint`
when the gateway is not reachable on the Docker daemon host.
- Podman Machine uses gvproxy's host-loopback route on macOS. Native Linux
Podman uses the primary loopback endpoint.
- Gateway restarts recover persisted objects from storage, but live relay
streams must be re-established by supervisors.
- User-facing behavior changes must update published docs in `docs/`; this file
Expand Down
2 changes: 1 addition & 1 deletion architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ The canonical main process receives the declared workload environment before
supervisor-only values are stripped and provider placeholders are injected.
Template environment is treated like user-provided sandbox environment. It can
shape the workload child, but it cannot override driver-controlled identity,
gateway callback, TLS, relay socket, proxy, provider, or supervisor coordination
gateway endpoint, TLS, relay socket, proxy, provider, or supervisor coordination
variables. Drivers and the supervisor rewrite those reserved values after image
and template environment are considered.

Expand Down
52 changes: 0 additions & 52 deletions crates/openshell-core/src/driver_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,58 +7,6 @@ use std::path::{Path, PathBuf};

use crate::proto::compute::v1::DriverSandbox;

/// Built-in sandbox network callback routes used to derive a callback endpoint
/// when an operator does not configure a per-driver `grpc_endpoint` override.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum GatewayCallbackRoute {
/// A Docker container reaches the host through Docker's gateway alias.
Docker,
/// A Podman container reaches the host through Podman's gateway alias.
Podman,
/// A libkrun guest reaches the host through gvproxy's gateway alias.
Vm,
}

/// Build the endpoint a sandbox uses to call its gateway for a known route.
///
/// The result is deliberately derived by the gateway rather than baked into
/// individual driver defaults. A configured `grpc_endpoint` remains an
/// operator override for remote or non-standard deployments.
#[must_use]
pub fn gateway_callback_endpoint(
route: GatewayCallbackRoute,
gateway_port: u16,
gateway_tls_enabled: bool,
) -> String {
let scheme = if gateway_tls_enabled { "https" } else { "http" };
let host = match route {
GatewayCallbackRoute::Docker | GatewayCallbackRoute::Vm => "host.openshell.internal",
GatewayCallbackRoute::Podman => "host.containers.internal",
};
format!("{scheme}://{host}:{gateway_port}")
}

#[cfg(test)]
mod callback_endpoint_tests {
use super::{GatewayCallbackRoute, gateway_callback_endpoint};

#[test]
fn derives_endpoint_for_each_builtin_route() {
assert_eq!(
gateway_callback_endpoint(GatewayCallbackRoute::Docker, 17670, false),
"http://host.openshell.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackRoute::Podman, 17670, true),
"https://host.containers.internal:17670"
);
assert_eq!(
gateway_callback_endpoint(GatewayCallbackRoute::Vm, 17670, true),
"https://host.openshell.internal:17670"
);
}
}

// ---------------------------------------------------------------------------
// Sandbox container/pod label keys (openshell.ai/ namespace)
// ---------------------------------------------------------------------------
Expand Down
29 changes: 16 additions & 13 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ mediates every supported TCP and DNS operation, attributes it to the calling
binary, and sends the request across the private channel. The supervisor
authorizes the request before it opens an upstream connection. Docker's absent
workload network is the mandatory outer fence if mediation fails or is
bypassed. Only the trusted supervisor companion joins the driver-owned bridge,
where it originates approved egress and can resolve other services on that
network.
bypassed. The trusted supervisor companion uses Docker host networking, where
it reaches the gateway's primary loopback listener and originates approved
egress.

The driver copies trusted runtime bytes from the configured supervisor image
through the Docker archive API. No workload launch depends on a host bind
Expand Down Expand Up @@ -73,7 +73,7 @@ LSM decisions remain authoritative.
| `cap_drop = ALL`, no `cap_add`, no-new-privileges | Prevents either container from acquiring Linux capabilities. |
| Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. |
| `network_mode = none` on the workload | Removes direct external routes. |
| Driver-owned bridge on the supervisor | Lets the trusted supervisor originate approved gateway and upstream connections and use Docker service discovery. |
| `network_mode = host` on the supervisor | Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. |
| `restart_policy = no` | Keeps canonical main-process exit terminal. |
| `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. |
| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. |
Expand Down Expand Up @@ -128,22 +128,25 @@ dynamically linked glibc `/openshell-supervisor` binary that runs in the
host-networked supervisor container. Release and gateway image builds bake
matching image tags into the binary.

## Callback and TLS
## Gateway session and TLS

`OPENSHELL_ENDPOINT` and gateway authentication material are injected only into
the supervisor companion. The workload never receives the sandbox JWT, gateway
client TLS key, policy authority, or interception CA private key.

When no endpoint is configured, the driver derives
`host.openshell.internal:<gateway-port>`. Native Linux uses the managed bridge
gateway. Docker Desktop and compatible VM-backed daemons use Docker's
`host-gateway` route. A configured HTTPS server certificate must include the
When no endpoint is configured, the supervisor connects to
`127.0.0.1:<gateway-port>`. Set `grpc_endpoint` when the gateway is not on the
Docker daemon host. A configured HTTPS server certificate must include the
endpoint host in its subject alternative names.

The driver pins a concrete managed-bridge address in the sandbox descriptor.
For Docker's special `host-gateway` route, it leaves the address unpinned so
the supervisor resolves the driver-injected alias in its own container. This
avoids treating the Docker VM's loopback as the desktop host.
The driver publishes host loopback as the backend address for
`host.openshell.internal`. Policy DNS resolves that reserved name through the
mediated path, so policies can reach host services without a Docker bridge,
container DNS alias, or another gateway listener.

Docker Engine on Linux supports host networking directly. Docker Desktop
requires host networking to be enabled in Settings and does not support it
when Enhanced Container Isolation is enabled.

The supervisor owns these security-critical variables:

Expand Down
Loading
Loading