Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,12 +297,15 @@ sandbox and can negate OpenShell workspace isolation and filesystem-policy
controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved.

Network features follow the driver/substrate split. Drivers own only the outer
fence and protected channel. The sandbox owns seccomp notification, local DNS,
socket virtualization, process observation, and binary identity. The supervisor
owns DNS eligibility, policy authorization, destination filtering, upstream
dials, relay behavior, credential rewriting, and OCSF decisions. No supported
path requires nftables, a workload network namespace, proxy environment
variables, added capabilities, or an unconfined AppArmor profile.
fence and protected channel. The native Linux adapter owns seccomp notification,
local DNS, socket virtualization, process observation, and binary identity. The
Kubernetes gVisor adapter instead combines the sentry and zero-rule workload
`NetworkPolicy` with a workload-local explicit proxy whose streams cross the
protected channel. The supervisor owns DNS eligibility, policy authorization,
destination filtering, upstream dials, relay behavior, credential rewriting,
and OCSF decisions. Native mode does not require proxy environment variables;
gVisor mode injects loopback HTTP proxy variables and applies endpoint-only
policy without added capabilities or an unconfined AppArmor profile.

The Kubernetes deployment packaging has two ownership boundaries. The gateway
chart owns the gateway workload, configuration, Services, PKI, and
Expand Down
18 changes: 14 additions & 4 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ Each sandbox has three trust levels:
| Component | Role |
|---|---|
| Supervisor | Owns gateway credentials, admitted policy, L7 proxying, SSH, and gateway relays. It never executes inside the agent workload. |
| Sandbox | Runs as the same non-root identity as the agent, installs the workload seccomp listener, applies the Landlock baseline, owns child processes, and mediates the protected supervisor channel. |
| Agent child | Inherits the sandbox network listener and runs with zero capabilities, `no_new_privs`, Landlock, and the final syscall filter. |
| Sandbox | Runs as the same non-root identity as the agent, owns child processes, mediates the protected supervisor channel, and instantiates the driver-selected native Linux or gVisor adapter. |
| Agent child | Runs with zero capabilities behind the selected adapter. Native Linux children inherit seccomp mediation, `no_new_privs`, Landlock, and the final syscall filter; gVisor children rely on the sentry, OCI mounts, and the outer network fence. |

The runtime grants neither trusted component nor agent child any Linux
capability inside the workload. Drivers resolve one exact non-root UID, GID,
Expand Down Expand Up @@ -71,8 +71,8 @@ replacement from granting authority.
run untrusted code yet.
3. `openshell-supervisor` loads policy and runtime settings from the gateway,
attaches to the sandbox, and verifies the driver's generation and evidence.
4. The sandbox installs its seccomp notification broker and Landlock baseline,
validates its mechanism-specific audit evidence, and reports backend-neutral
4. The sandbox starts the selected runtime adapter, validates its mechanism-
specific audit evidence, and reports backend-neutral
enforcement properties. The supervisor must accept those properties and
their immutable session and resource binding before it sends the launch
permit. Other isolation backends may establish the same properties with
Expand Down Expand Up @@ -131,6 +131,16 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. |
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |

The Kubernetes gVisor adapter uses the same authenticated lifecycle and process
backend with different enforcement mechanisms. The gVisor sentry and OCI mounts
provide the workload boundary. A zero-rule Kubernetes egress `NetworkPolicy`
blocks direct connections, while a workload-local HTTP/CONNECT listener reverse-
tunnels streams to the existing supervisor proxy. The supervisor loads policy
in endpoint-only mode because the tunnel authenticates the sandbox generation,
not an individual executable. This adapter intentionally does not claim native
Landlock path policy, nested child seccomp, transparent TCP, or per-binary
network attribution.

The supervisor may enrich baseline filesystem allowances for runtime-required
paths, such as proxy support files or GPU device paths when a GPU is present.
These internal allowances must stay sandbox-scoped and avoid exposing host
Expand Down
6 changes: 4 additions & 2 deletions crates/openshell-driver-docker/src/isolation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ use openshell_isolation_interface::contract::{
};
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeAdapter,
SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
};
use serde::Serialize;

Expand Down Expand Up @@ -119,6 +119,7 @@ impl DockerBoundarySpec {
resource_claims: resource_claims.clone(),
resource_claim_files: BTreeMap::new(),
workload_identity: self.workload_identity.clone(),
adapter: SandboxRuntimeAdapter::default(),
outer_fence: outer_fence.clone(),
child_env: self.child_env,
},
Expand All @@ -127,6 +128,7 @@ impl DockerBoundarySpec {
generation: self.generation,
session_id: self.session_id,
workload_identity: self.workload_identity,
adapter: SandboxRuntimeAdapter::default(),
transport: SandboxTransport::Unix {
socket_path: self.control_socket,
},
Expand Down
10 changes: 10 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,16 @@ default seccomp profile. The sandbox installs a nested seccomp user-notification
filter without requesting a capability in the Pod spec. Startup fails closed
when the runtime blocks the required seccomp or Landlock operations.

An effective `pod.runtime_class_name` of `gvisor` selects the gVisor adapter in
the same sandbox backend. The workload Pod keeps the non-root, drop-all-
capabilities posture but omits the Kubernetes seccomp profile and custom sysctl
that GKE Sandbox does not support. The sandbox qualifies the gVisor sentry,
starts an explicit proxy on `127.0.0.1:3128`, and reverse-tunnels proxy streams
to the supervisor over the authenticated boundary protocol. The empty-egress
`NetworkPolicy` blocks direct workload connections. This mode applies endpoint-
only network policy and does not apply Landlock path rules, the nested child
seccomp filter, transparent TCP, or per-binary network attribution.

The supervisor Pod has a direct, non-controller owner reference to the Sandbox
resource. This links its garbage-collection lifecycle to the sandbox without
competing with the Agent Sandbox controller for workload-Pod ownership.
Expand Down
145 changes: 125 additions & 20 deletions crates/openshell-driver-kubernetes/src/driver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1962,6 +1962,7 @@ impl KubernetesComputeDriver {
return Err(fail("pair labels changed"));
}
let spec = pod.spec.as_ref().ok_or_else(|| fail("missing Pod spec"))?;
let gvisor = spec.runtime_class_name.as_deref() == Some("gvisor");
if spec.host_network == Some(true)
|| spec.host_pid == Some(true)
|| spec.host_ipc == Some(true)
Expand Down Expand Up @@ -2013,10 +2014,12 @@ impl KubernetesComputeDriver {
.supplemental_groups
.as_deref()
.is_some_and(|groups| !groups.is_empty())
|| security
.seccomp_profile
.as_ref()
.is_none_or(|profile| profile.type_ != "RuntimeDefault")
|| (!gvisor
&& security
.seccomp_profile
.as_ref()
.is_none_or(|profile| profile.type_ != "RuntimeDefault"))
|| (gvisor && security.seccomp_profile.is_some())
{
return Err(fail("numeric identity, groups, or seccomp profile changed"));
}
Expand Down Expand Up @@ -2046,9 +2049,17 @@ impl KubernetesComputeDriver {
&& sysctl.get("value").and_then(serde_json::Value::as_str) == Some("0")
})
});
if !unprivileged_port_sysctl {
if !gvisor && !unprivileged_port_sysctl {
return Err(fail("safe unprivileged-port sysctl changed"));
}
if gvisor
&& pod_json
.pointer("/spec/securityContext/sysctls")
.and_then(serde_json::Value::as_array)
.is_some_and(|sysctls| !sysctls.is_empty())
{
return Err(fail("gVisor workload must not request custom sysctls"));
}
let check_container = |container: &k8s_openapi::api::core::v1::Container,
name: &str|
-> Result<(), KubernetesDriverError> {
Expand Down Expand Up @@ -2265,6 +2276,16 @@ impl KubernetesComputeDriver {
agent_gid,
&names.sandbox_secret,
)?;
let runtime_adapter = if workload_pod
.spec
.as_ref()
.and_then(|spec| spec.runtime_class_name.as_deref())
== Some("gvisor")
{
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
} else {
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
};
let workload_pod_uid =
workload_pod.metadata.uid.clone().ok_or_else(|| {
KubernetesDriverError::Message("workload Pod has no UID".to_string())
Expand Down Expand Up @@ -2308,6 +2329,22 @@ impl KubernetesComputeDriver {
child_env.extend(spec.environment.clone());
}
child_env.retain(|name, _| !name.starts_with("OPENSHELL_"));
if matches!(
runtime_adapter,
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
) {
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
}
child_env.insert(
"NO_PROXY".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
child_env.insert(
"no_proxy".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
}
let host_gateway_ip = self.config.host_gateway_ip.parse().ok();
let session_id = launch_authentication.supervisor.session_id;
let tls = generate_sandbox_tls_material(session_id)
Expand Down Expand Up @@ -2368,6 +2405,7 @@ impl KubernetesComputeDriver {
},
host_gateway_ip,
workload_identity,
adapter: runtime_adapter,
child_env,
}
.provision()
Expand Down Expand Up @@ -2471,7 +2509,7 @@ impl KubernetesComputeDriver {
supervisor_uid: &str,
agent_uid: u32,
agent_gid: u32,
child_env: std::collections::HashMap<String, String>,
mut child_env: std::collections::HashMap<String, String>,
launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication,
) -> Result<(), KubernetesDriverError> {
let namespace_uid = Api::<Namespace>::all(self.client.clone())
Expand Down Expand Up @@ -2526,6 +2564,32 @@ impl KubernetesComputeDriver {
agent_gid,
&names.sandbox_secret,
)?;
let runtime_adapter = if workload_pod
.spec
.as_ref()
.and_then(|spec| spec.runtime_class_name.as_deref())
== Some("gvisor")
{
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
} else {
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
};
if matches!(
runtime_adapter,
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
) {
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
}
child_env.insert(
"NO_PROXY".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
child_env.insert(
"no_proxy".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
}
let workload_pod_uid =
workload_pod.metadata.uid.clone().ok_or_else(|| {
KubernetesDriverError::Message("workload Pod has no UID".to_string())
Expand Down Expand Up @@ -2607,6 +2671,7 @@ impl KubernetesComputeDriver {
},
host_gateway_ip: self.config.host_gateway_ip.parse().ok(),
workload_identity,
adapter: runtime_adapter,
child_env,
}
.provision()
Expand Down Expand Up @@ -5217,6 +5282,10 @@ fn apply_supervisor_sandbox_runtime_boundary(
else {
return;
};
let gvisor = spec
.get("runtimeClassName")
.and_then(serde_json::Value::as_str)
== Some("gvisor");
spec.insert("hostNetwork".to_string(), serde_json::json!(false));
spec.insert("hostPID".to_string(), serde_json::json!(false));
spec.insert("hostIPC".to_string(), serde_json::json!(false));
Expand All @@ -5240,20 +5309,22 @@ fn apply_supervisor_sandbox_runtime_boundary(
]
}),
);
spec.insert(
"securityContext".to_string(),
serde_json::json!({
"runAsUser": params.sandbox_uid,
"runAsGroup": params.sandbox_gid,
"runAsNonRoot": true,
"fsGroup": params.sandbox_gid,
"fsGroupChangePolicy": "OnRootMismatch",
"supplementalGroups": [],
"supplementalGroupsPolicy": "Strict",
"seccompProfile": {"type": "RuntimeDefault"},
"sysctls": [{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}]
}),
);
let mut pod_security = serde_json::json!({
"runAsUser": params.sandbox_uid,
"runAsGroup": params.sandbox_gid,
"runAsNonRoot": true,
"fsGroup": params.sandbox_gid,
"fsGroupChangePolicy": "OnRootMismatch",
"supplementalGroups": [],
"supplementalGroupsPolicy": "Strict"
});
if !gvisor {
pod_security["seccompProfile"] = serde_json::json!({"type": "RuntimeDefault"});
pod_security["sysctls"] = serde_json::json!([
{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}
]);
}
spec.insert("securityContext".to_string(), pod_security);
let volumes = spec
.entry("volumes")
.or_insert_with(|| serde_json::json!([]))
Expand Down Expand Up @@ -8505,6 +8576,40 @@ mod tests {
);
}

#[test]
fn gvisor_runtime_omits_incompatible_kernel_security_context() {
let pod_template = {
let params = SandboxPodParams {
default_runtime_class_name: "gvisor",
..SandboxPodParams::default()
};
sandbox_template_to_k8s(
&SandboxTemplate::default(),
false,
&std::collections::HashMap::new(),
true,
&params,
)
};

assert_eq!(
pod_template["spec"]["runtimeClassName"],
serde_json::json!("gvisor")
);
assert!(
pod_template["spec"]["securityContext"]["seccompProfile"].is_null(),
"GKE Sandbox rejects Kubernetes seccomp profiles"
);
assert!(
pod_template["spec"]["securityContext"]["sysctls"].is_null(),
"GKE Sandbox rejects custom sysctls"
);
assert_eq!(
pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["drop"],
serde_json::json!(["ALL"])
);
}

#[test]
fn template_runtime_class_name_overrides_config_default() {
let template = SandboxTemplate {
Expand Down
Loading
Loading