feat(sandbox,podman): trust corporate CA for https:// proxies and intercepted TLS

[feloy]

Summary

Adds support for https:// corporate egress proxies and for trusting an operator-provided corporate CA bundle, so sandboxes can reach a forward proxy that terminates TLS with a private CA and can operate behind a TLS-intercepting proxy (mitmproxy, squid ssl-bump) that re-signs tunneled server certificates. The feature is delivered through the hardened, driver-supplied argument model — never environment variables — and is fail-closed.

Related Issue

Refs #1792

Changes

• Shared URL validator (openshell-core): parse_upstream_proxy_url now accepts https:// (recording the scheme via a new UpstreamProxyAddr.secure), keeping the explicit-port requirement; adds the PROXY_CA_MOUNT_PATH container mount constant.
• Supervisor (openshell-supervisor-network):
  • ProxyEndpoint gains a TLS client config; for an https:// proxy the connection to the proxy is wrapped in TLS (verifying the proxy certificate) before the CONNECT handshake.
  • PrefixedStream now wraps a Plain/Tls UpstreamStream enum so the relay paths consume either transport transparently.
  • The corporate CA bundle is read fail-closed (unreadable or certificate-free file is fatal) and reuses the existing build_upstream_client_config; tls_connect_upstream was already generic, so l7/tls.rs is unchanged.
  • run.rs folds the corporate CA into the sandbox combined trust bundle (write_ca_files) and the L7 upstream verification store, so intercepted upstream handshakes succeed and sandbox workloads trust the re-signed certificates.
• Config surface: new --upstream-proxy-ca-bundle supervisor argument (no env =); new Podman driver proxy_ca_bundle setting (TOML key, --sandbox-proxy-ca-bundle, OPENSHELL_SANDBOX_PROXY_CA_BUNDLE). The driver validates the pairing (a CA bundle requires a proxy URL), bind-mounts the host PEM read-only into the sandbox (a CA certificate is not secret, so a plain bind mount rather than a driver secret), and performs a create-time readability check.
• Dev tooling: the standalone gateway task (tasks/scripts/gateway.sh) passes OPENSHELL_SANDBOX_PROXY_CA_BUNDLE through to the generated Podman config, so a local gateway can be pointed at a TLS-intercepting proxy without hand-editing the regenerated TOML.
• Docs: updated docs/reference/gateway-config.mdx, the Podman driver README.md, and the architecture/sandbox.md trust-model section.

Testing

• mise run pre-commit passes
• Unit tests added/updated — URL parsing (http/https, socks rejected), TLS-wrapped CONNECT against a fake TLS proxy (trusted-CA success and untrusted-cert rejection), fail-closed CA-bundle validation, and Podman driver config validation + bind-mount/argv assertions.
• E2E tests added/updated — new serialized podman_corporate_proxy case drives an https:// proxy end to end: the fake proxy self-signs, exposes its CA, and that CA is fed back via proxy_ca_bundle. (Runs only in the Podman e2e job.)
• Manually verified against a real TLS-intercepting proxy — mitmproxy on the Podman sandbox network, reached as an https:// proxy with its CA supplied via proxy_ca_bundle. This exercises both halves in one run: the TLS-wrapped CONNECT to the proxy listener, and the upstream certificate that mitmproxy re-signs with the same CA.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[derekwaynecarr]

Rather than make a change for just podman, can you explore having parity across all compute drivers, or note why its not possible before we merge this?

[feloy]

Rather than make a change for just podman, can you explore having parity across all compute drivers, or note why its not possible before we merge this?

My idea would be to implement by pieces as small as possible. The first part (#2245) is connection to proxy with http only and Podman only. This PR is connection to proxy with https and certificate, for Podman only. The follow-up PRs would be the full stack (http+https) for each other driver. For Docker, it should be very similar, but I still didn't investigate what would be needed for others drivers.

[kevin-pedretti]

Just a quick note to say I tested this PR and it worked in my environment. This got things working with my proxy that does SSL interception.

It may be useful to separate the host path and gateway container paths so they don't need to match. For example, something like:

  [podman]
  proxy_ca_bundle = "/etc/openshell/certs/ca-bundle.pem"  # Path for gateway to validate
  proxy_ca_bundle_host_path = "/home/myuser/ca-bundle.pem"  # Path for host Podman

  # If host_path is unset, use proxy_ca_bundle (backwards compatible)

With the current setup, I had to be careful to make sure the host and container paths to ca-bundle.pem matched. Otherwise sandbox creation fails when the gateway tries to validate the (single) path specified by proxy_ca_bundle (a host path).

[feloy]

Just a quick note to say I tested this PR and it worked in my environment. This got things working with my proxy that does SSL interception.

It may be useful to separate the host path and gateway container paths so they don't need to match. For example, something like:

  [podman]
  proxy_ca_bundle = "/etc/openshell/certs/ca-bundle.pem"  # Path for gateway to validate
  proxy_ca_bundle_host_path = "/home/myuser/ca-bundle.pem"  # Path for host Podman

  # If host_path is unset, use proxy_ca_bundle (backwards compatible)

With the current setup, I had to be careful to make sure the host and container paths to ca-bundle.pem matched. Otherwise sandbox creation fails when the gateway tries to validate the (single) path specified by proxy_ca_bundle (a host path).

@kevin-pedretti thanks for testing this PR.

I don't think this change is necessary: proxy_ca_bundle is where the ca bundle is found in your host ("/home/myuser/ca-bundle.pem" in your case) , and the ca bundle will be mounted in an hardcoded path in the container ("/etc/openshell/tls/proxy/ca-bundle.pem").

[johntmyers]

gator-agent

PR Review Status

Validation: This PR is project-valid as a focused follow-up to the already-triaged corporate-proxy gap in #1792 and the earlier HTTP proxy work in #2245. Thanks @derekwaynecarr — I checked the driver-parity concern you raised. @feloy explained that this is an incremental Podman delivery with other drivers planned as follow-ups; the supervisor portion is shared, so that scope choice does not prevent code review, while maintainers retain the final scope decision. I also checked @feloy's explanation of the CA path behavior: the configured host path is mounted at a fixed container path, matching the current driver implementation.
Head SHA: a6b67a548178b5f4a65aeaf7e93adef947a53c49
Base SHA: 877ddbacb4b915fa1d5bc6f02302d9a5c13a18e1
Merge base SHA: 877ddbacb4b915fa1d5bc6f02302d9a5c13a18e1
Patch ID: f7885d7f4a97aabbe6342feda4064351f7b4d5ef
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

• GATOR-a6b67a54-01: the corporate CA validation counts decoded PEM blocks without checking whether rustls accepts any certificate as a trust anchor, so a malformed bundle can pass startup and fail later despite the documented fail-closed behavior. See the inline finding.

Carried findings:

• None

Non-blocking suggestions:

• For an https:// proxy, consider allowing proxy_auth_file without proxy_auth_allow_insecure; the credential is sent only after the verified TLS connection, so the current plaintext-risk acknowledgement and diagnostic are misleading for this newly supported path. Relevant sites: crates/openshell-driver-podman/src/config.rs:368, crates/openshell-supervisor-network/src/upstream_proxy.rs:483, and crates/openshell-driver-podman/README.md:370.

Docs: Fern reference docs and the Podman/architecture documentation are updated.

Next state: gator:in-review

[johntmyers]

gator-agent

Warning — GATOR-a6b67a54-01

Invariant: A configured corporate CA bundle must contain at least one X.509 certificate that rustls actually accepts as a trust anchor; syntactically valid PEM framing with invalid DER must not satisfy fail-closed validation and silently fall back to other roots.

Prerequisite: A Podman gateway operator configures proxy_ca_bundle with a non-empty regular file whose PEM block decodes but whose DER is not a valid X.509 trust anchor, such as a truncated or incorrectly generated bundle.

Entry point → sink: documented [openshell.drivers.podman] proxy_ca_bundle with a supported corporate proxy → UpstreamProxyConfig::from_args accepts it, then build_upstream_client_config ignores the unusable certificate and the first private-CA proxy or intercepted-upstream TLS handshake fails.

Base → head: the base had no CA-bundle input and rejected https:// proxy URLs → this head counts every successfully base64-decoded CERTIFICATE block with rustls_pemfile::certs(...).flatten() even when RootCertStore later rejects its DER; built-in roots keep the aggregate store non-empty, so startup succeeds.

Impact: A documented present-but-invalid setting is accepted rather than failing closed. Sandboxes start, but corporate TLS egress then fails at runtime with an opaque outage; a publicly trusted proxy listener can also operate although the explicitly configured corporate bundle contributed no usable anchor.

Reproducer: In the upstream_proxy unit module, install the ring provider, write -----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n to a temporary file, and construct config with HTTPS_PROXY=https://proxy.corp.com:3130 and PROXY_CA_BUNDLE=<temp path>. Head returns Ok; the expected deterministic result is an error naming PROXY_CA_BUNDLE and reporting zero usable certificates/trust anchors.

PR ownership: This PR introduces proxy_ca_bundle, documents unreadable or certificate-free bundles as fatal, and adds this insufficient count check; the misleading acceptance does not exist at base. The same helper is reread from run.rs:223, so both proxy-listener and intercepted-upstream paths share the invariant.

Requested change: Validate decoded certificates with the same rustls RootCertStore acceptance path used for TLS and reject the bundle unless at least one certificate is added successfully. Mixed bundles may remain tolerant, but add the invalid-DER reproducer so zero accepted anchors fail at supervisor startup.