Skip to content

fix(podman): resolve local callbacks to loopback on Linux - #3435

Open
politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:3412-podman-multihomed-callback-loopback/qb
Open

politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:3412-podman-multihomed-callback-loopback/qb

Conversation

@politerealism

Copy link
Copy Markdown
Contributor

Summary

On Linux, the Podman driver's automatic local-callback path negotiated a bridge-network gateway address (rootful) or the host's default-route interface (rootless pasta), and aliased host.containers.internal/host.openshell.internal through Podman's host-gateway resolver. Since RFC 0012 (#2942), the callback-capable supervisor always runs in the host network namespace and shares loopback with the gateway, so on multi-homed hosts these two independent interface selections can disagree, leaving the supervisor unable to reach the gateway.

Related Issue

Fixes #3412

Changes

  • driver.rs: automatic Linux callback path now always requests a loopback gateway listener, for both rootful and rootless Podman; removed the now-obsolete bridge-gateway-IP discovery and rootless-pasta-only gating. Explicit host_gateway_ip override path is simplified and now works uniformly regardless of rootless network helper.
  • container.rs: host aliases resolve to 127.0.0.1 directly on Linux instead of Podman's host-gateway magic value, matching the requested loopback listener. macOS Podman Machine behavior (gvproxy host-loopback forwarding) is unchanged.
  • client.rs: removed the now-dead network_gateway_ip bridge-inspection method (no remaining callers).
  • config.rs, NETWORKING.md, architecture/gateway.md, docs/reference/gateway-config.mdx, docs/reference/sandbox-compute-drivers.mdx, skills/debug-openshell-cluster/SKILL.md: updated to describe the new loopback-based default and troubleshooting guidance.

Out of scope (left for follow-up, per the issue's own implementation plan): removing PR #2874's delayed-bind protobuf/server machinery (a separate, wire-breaking proto cleanup), and the misleading ContainerExited: code 0 diagnostic when the supervisor exits before the workload.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated (crates/openshell-driver-podman/src/driver.rs, container.rs) — 224 passing, including new/updated coverage for: rootful and rootless automatic loopback selection, explicit override still works with any rootless helper, constructor no longer inspects the bridge network for local callbacks.
  • cargo fmt --all -- --check and cargo clippy -p openshell-driver-podman --all-targets -- -D warnings clean
  • mise run markdown:lint clean
  • E2E tests added/updated (not applicable — no existing multi-homed-host CI lane to extend; issue reproduction requires a real multi-interface host)

Checklist

Since RFC 0012 (NVIDIA#2942), the callback-capable supervisor always runs in
the host network namespace, for both rootful and rootless Podman. The
driver still requested a callback listener via bridge-network gateway
inspection (rootful) or default-route negotiation (rootless pasta),
and aliased host.containers.internal/host.openshell.internal through
Podman's host-gateway resolver. On multi-homed hosts, Podman's
host-gateway resolution and the driver's own negotiation can each pick
a different interface than the one the supervisor actually shares with
the gateway (loopback), leaving sandbox creation unreachable.

Request loopback directly for the automatic local-callback path on
Linux, and resolve the host aliases to 127.0.0.1 to match, regardless
of rootful/rootless mode or which rootless network helper Podman
reports. An explicit host_gateway_ip override is unaffected. macOS
Podman Machine behavior (gvproxy host-loopback forwarding) is
unchanged.

Fixes NVIDIA#3412

Signed-off-by: politerealism <burdcat17@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@drew

drew commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

With the new split, I think I we can actually remove all the callback machinery and remove this class of bug. Could you see if #3365 resolve your issue?

@politerealism

Copy link
Copy Markdown
Contributor Author

Manual verification

Beyond the unit tests, I ran this against a real local Podman-backed gateway built from this branch to confirm the fix end-to-end.

Test 1 — default config (loopback-bound gateway):

$ openshell sandbox create --name loopback-test ...
Phase: Ready
Conditions: Ready: True (DependenciesReady) - Supervisor session connected

Inspected the supervisor container directly:

$ podman inspect openshell-supervisor-<id> --format '{{json .HostConfig.ExtraHosts}}'
["host.containers.internal:127.0.0.1","host.openshell.internal:127.0.0.1"]

Test 2 — gateway primary listener bound to a real interface IP (172.16.21.61), reproducing the topology from the bug report where the primary listener doesn't cover loopback:

Gateway log:

Gateway listener bound address=172.16.21.61:18080 listener_purpose="primary"
Gateway listener bound address=127.0.0.1:18080 listener_purpose="compute-driver-callback" driver=podman reason="Podman supervisor callback shares the host network namespace with the gateway"

The reason string is the exact string from the new gateway_listener_requirements() path in driver.rs, confirming the LoopbackInterface selector is what's actually exercised (not the old bridge-gateway/default-route negotiation).

Sandbox creation against that gateway also reached Ready:

$ openshell sandbox create --gateway-endpoint http://172.16.21.61:18080 --name loopback-test-2 ...
Phase: Ready

Supervisor container again showed host.containers.internal:127.0.0.1 / host.openshell.internal:127.0.0.1, NetworkMode: host.

Cleaned up both test sandboxes and the local gateway afterward; no leftover containers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Podman driver picks the wrong host IP for the supervisor callback on multi-homed Linux hosts (regression from #2942)

2 participants