fix(ci)!: remove gateway callback listener dependency

[drew]

Summary

Fix the release canary regressions exposed after RFC 0012 landed. Docker supervisors now use host networking and the operator-configured primary gateway endpoint, so compute drivers no longer need to request extra gateway callback listeners.

Related Issue

No issue required: this is a focused follow-up for failures in the post-merge Release Canary on PR #3360.

Changes

• Use canary sandbox names that fit the 19-character API limit.
• Acknowledge enforced Kubernetes NetworkPolicy in the canary Helm install.
• Run Docker supervisor companion containers with host networking and connect them to the primary gateway endpoint.
• Remove compute-driver callback-listener negotiation, callback-only routing scopes, and nonlocal bind support.
• Document that drivers cannot request additional gateway listeners.
• Update the local Docker launcher to build and configure the supervisor and sandbox runtime images, without staging the old combined binary.
• Keep release-canary and cluster-debugging contributor instructions aligned with the runtime behavior.

This intentionally breaks the compute-driver protobuf API by removing GetGatewayListenerRequirements and GatewayListenerRequirement. External drivers must regenerate their bindings and connect supervisors to the configured primary gateway endpoint.

Testing

• mise run pre-commit
• cargo test -p openshell-server gateway_listener --lib
• cargo test -p openshell-server storage_proto --lib
• cargo test -p openshell-driver-docker
• cargo test -p openshell-driver-podman
• cargo test -p openshell-driver-vm --lib
• Docker E2E callback-sensitive paths: create, exec, delete, stop/start, gateway restart, policy updates, provider access, loopback forwarding
• Local Docker launcher smoke test: gateway start, sandbox create, exec, and delete
• Full Docker E2E: local run reached an unrelated provider_readiness fixture-image build failure while extending the distroless supervisor image
• Remote driver E2E and Release Canary

Checklist

• Follows Conventional Commits
• Commit is signed off (DCO)

[github-actions]

Label test:e2e-kubernetes applied for 9ce3e82. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway, sandbox, and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

[github-actions]

Label test:e2e applied for 9ce3e82. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

/ok to test 9ce3e82

[drew]

/ok to test 0695642

[drew]

/ok to test 098ae15

[drew]

/ok to test b143c77

[drew]

/ok to test 3de674b

[drew]

Validation complete for 3de674b:

• Branch Checks: passed
• Branch E2E: passed (Docker, Podman, Kubernetes, managed VM, and external-driver paths)
• Helm Lint and workflow security checks: passed
• Manual Release Canary: Ubuntu Docker, Kubernetes Helm, and macOS Homebrew passed

The manual Snap and Fedora canary jobs used the already-published pre-fix artifacts from Release Dev run 35055785361, so they reproduce the two failures fixed by this PR: missing Snap JWT bootstrap material and the pre-IP_FREEBIND Podman callback listener. The Snap wrapper has focused regression coverage, and the Fedora listener path is covered by the new Linux exact-address bind test plus the passing Podman E2E suite.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3365.docs.buildwithfern.com/openshell

[krishicks]

I like the design direction on this. The agent had this to say about test coverage:

The Podman endpoint tests duplicate stale endpoint-selection logic rather than
calling production code. On Linux, production selects 127.0.0.1, while the tests
assert host.containers.internal, yet still pass.

Recommended fix:

 1. Extract endpoint selection into a helper.
 2. Call it from PodmanComputeDriver::new.
 3. Test Linux and non-Linux behavior through that helper.
 4. Keep the explicit-override test.

The other things it found were related to breaking changes, which we expect. Issues and their workarounds:

• External compute-driver authors: Remove GetGatewayListenerRequirements support and rebuild against the updated protobuf API.
• Operators: Configure grpc_endpoint so supervisors can reach the gateway’s primary listener.
• Docker operators: Remove network_name and host_gateway_ip from Docker configuration.
• Docker Desktop users: Enable host networking and disable Enhanced Container Isolation.
• TLS users: Ensure the configured grpc_endpoint appears in the gateway certificate SANs.

[drew]

Fixed in e924ebd. Podman now routes both production initialization and tests through select_grpc_endpoint: Linux selects 127.0.0.1, Podman Machine selects host.containers.internal, TLS still selects the scheme, and an explicit override wins. cargo test -p openshell-driver-podman --lib and mise run pre-commit pass.

[elezar]

Non-blocking observation: before this PR, remote external drivers could use a callback-only listener that rejected non-supervisor APIs at the listener boundary. With this change, remote supervisors use the primary listener, which may require exposing the gateway’s full HTTP/gRPC surface beyond loopback. Sandbox JWT authorization still restricts supervisor credentials, but this removes a defense-in-depth boundary and may increase exposure to authentication regressions, accidentally unauthenticated endpoints, or pre-authentication denial-of-service traffic. This seems worth recognizing as a security tradeoff, though I don’t consider it a blocker for this PR.