Skip to content

sdk/js: reach peers through a router the control plane names by /dnsaddr - #513

Merged
aojea merged 3 commits into
google:mainfrom
aojea:fix/sdk-js-dnsaddr-relay
Sep 26, 2026
Merged

aojea merged 3 commits into
google:mainfrom
aojea:fix/sdk-js-dnsaddr-relay

Conversation

@aojea

@aojea aojea commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

The js-calls-python-agent canary failed on deploys #403 and #404 with the last 200 bytes of a stack trace. This is the cause, the fix, and the tests that would have caught it before a deploy.

Cause

The testnets advertise their routers as /dnsaddr/bootstrap.<env>.sam-mesh.dev/p2p/<id> (router --external-addr). The JS SDK built relay addresses on the address the credential carried, so a call by peer ID dialed /dnsaddr/<host>/p2p/<router>/p2p-circuit/p2p/<peer>. js-libp2p 3.3.11 resolves such an address to the TXT records themselves and drops what follows the dnsaddr's /p2p/<router>; the dial queue then discards the records as the wrong peer and throws NoValidAddressesError. Every JS call by peer ID failed in the testnet. The Python SDK resolves the router address itself and dials the relay by peer ID, so python-calls-js-agent passed.

The integration harness's router advertises /ip4 addresses, so nothing exercised /dnsaddr; the images built from the same commits run the pair fine locally.

Fix

An admitted router is known by the resolved address its connection was made on (conn.remoteAddr), so a relay address never carries a /dnsaddr. A mesh host takes a dns resolver so the case is testable.

Tests

  • sdk/js/src/session.test.ts: joins through a router handed out as /dnsaddr/router.test/p2p/<id>, with a stub resolver answering the TXT lookup, and authenticates a peer through it. Without the fix it fails with NoValidAddressesError.
  • tests/integration/sdk_canary_test.go (TestSDKCanaryScript, ~5s): runs canary.sh from .github/k8s/sam-sdk-canary-template.yaml as the pod does — each language's agent example logging to the shared directory, the other language's A2A example as CALL, both enrolled with one token, the second call resuming the identity — and checks the verdict lines and the readiness file. It also feeds canned Node and Python failures and checks the error line reported.
  • The canary script reports a failed call by the line naming the error (Node prints it before the stack trace, Python after) and takes its directory from SAM_CANARY_DIR.

Lint

A second commit makes hack/lint.sh pass: golangci-lint v2.14.0 and deadcode from x/tools v0.50.0 (v0.40.0's SSA builder panics under Go 1.27, so the deadcode step never ran), the two remaining ReverseProxy.Director uses moved to Rewrite, reflect.Ptr → reflect.Pointer, and a go fmt fix.

Validation: npm run typecheck, npm test (66), TestNativeSDKsMesh|TestNativeSDKA2A|TestNativeSDKExamples|TestSDKCanaryScript, go build ./..., go test ./internal/node ./internal/storage ./internal/standalone, hack/lint.sh exit 0.

The testnets advertise their routers as
/dnsaddr/bootstrap.<env>.sam-mesh.dev/p2p/<id>. The JS SDK built relay
addresses on the address the credential carried, and js-libp2p resolves
/dnsaddr/<host>/p2p/<router>/p2p-circuit/p2p/<peer> to the TXT records
themselves, dropping the circuit, so every dial of a peer by ID failed
with NoValidAddressesError. The js-calls-python-agent canary reported
this on deploys 403 and 404 as the last 200 bytes of a stack trace. The
Python SDK resolves the router address itself and dials the relay by
peer ID, so python-calls-js-agent passed.

An admitted router is now known by the address its connection was made
on, resolved. A host takes a DNS resolver, and session.test.ts joins
through a router handed out as /dnsaddr, with a stub resolver answering
the TXT lookup, and reaches a peer through it; without the change the
test fails with NoValidAddressesError.

The canary script reports a failed call by the line that names the
error, which Node prints before the stack trace and Python after, and
takes its directory from SAM_CANARY_DIR. TestSDKCanaryScript runs the
script from the template as the pod does: each language's agent example
with its output in the shared directory, the other language's A2A
example as the caller, both enrolled with one token, the second call
resuming the identity, and a failing call reported by its error line.
x/tools v0.40.0's SSA builder panics on this tree under Go 1.27
("unexpected expr: *ast.KeyValueExpr"), so the deadcode step never ran.
golangci-lint v2.14.0 flags what v2.11.4 let through: ReverseProxy's
Director is deprecated since Go 1.26 and reflect.Ptr is an alias.

The egress proxy and the inference proxy use Rewrite. The inference
proxy's Rewrite is empty: ReverseProxy strips the inbound Forwarded and
X-Forwarded-* headers before calling it and blanks a missing User-Agent
itself, which is what the Director and the transport did by hand.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several updates, including migrating httputil.ReverseProxy from Director to Rewrite in inference_service.go and sidecar.go, upgrading linter versions in hack/lint.sh, adding support for /dnsaddr router resolution in the JS SDK, and introducing integration tests for the SDK canary script. Feedback on these changes highlights two main issues: first, the empty Rewrite function in inference_service.go fails to strip Forwarded and X-Forwarded-* headers or blank missing User-Agent headers, which requires explicit handling; second, the integration test in sdk_canary_test.go violates the repository's style guide by using an excessively long 60-second timeout instead of being tightly time-bounded.

Comment on lines +113 to +116
// The transport addresses the backend; the proxy itself strips the
// inbound Forwarded and X-Forwarded-* headers and blanks a missing
// User-Agent.
Rewrite: func(*httputil.ProxyRequest) {},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

5. API surfaces and secrets / General Correctness & Security

The empty Rewrite function does not automatically strip Forwarded or X-Forwarded-* headers because ProxyRequest.SetURL is not called here. Consequently, any client-supplied X-Forwarded-* headers will be forwarded to the backend LLM/inference service since the explicit Del calls were removed from RoundTrip.

Additionally, the previous Director implementation blanked a missing User-Agent to prevent Go's default User-Agent (e.g., Go-http-client/1.1) from being appended. This blanking behavior is now lost.

Please update the Rewrite function to explicitly delete these headers and blank the User-Agent if it is missing.

		// The transport addresses the backend; the proxy itself strips the
		// inbound Forwarded and X-Forwarded-* headers and blanks a missing
		// User-Agent.
		Rewrite: func(pr *httputil.ProxyRequest) {
			pr.Out.Header.Del("Forwarded")
			pr.Out.Header.Del("X-Forwarded-For")
			pr.Out.Header.Del("X-Forwarded-Host")
			pr.Out.Header.Del("X-Forwarded-Proto")
			if _, ok := pr.In.Header["User-Agent"]; !ok {
				pr.Out.Header.Set("User-Agent", "")
			}
		},

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not what ReverseProxy does. In ServeHTTP (Go 1.27, net/http/httputil/reverseproxy.go), when Rewrite is set the proxy deletes Forwarded, X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto from the outbound request before calling Rewrite, unconditionally; SetURL and SetXForwarded are what a Rewrite calls to add values back, and this one calls neither. The User-Agent blanking (if _, ok := outreq.Header["User-Agent"]; !ok { outreq.Header.Set("User-Agent", "") }) runs after both branches, so it applies in Rewrite mode as it did in Director mode; the Director's copy of it was redundant even before this change.

17ed29d adds TestInferenceService_ForwardingHeadersDoNotReachTheBackend, which sends the spoofed Forwarded/X-Forwarded-* headers and no User-Agent through svc.Handler() to a recording backend and asserts none of them arrive and the User-Agent is empty. With a Rewrite that copies the inbound headers through, the test fails on all four; with the empty Rewrite it passes.

Comment thread tests/integration/sdk_canary_test.go Outdated
Comment on lines +222 to +223
case <-time.After(60 * time.Second):
t.Errorf("%s: no verdict within 60s", canary)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

3. Test coverage must follow the pyramid

Under Rule 3, integration tests must be time-bounded and should not have long or unbounded waits. The repository style guide specifies that integration tests are bounded to 10 seconds per execution.

Using a 60-second timeout via time.After(60 * time.Second) is excessively long for a test that is expected to run in ~5 seconds with INTERVAL=1. Please reduce this timeout to a much shorter duration (e.g., 5 seconds) to ensure the test fails quickly if it hangs, adhering to the 10-second execution limit.

Suggested change
case <-time.After(60 * time.Second):
t.Errorf("%s: no verdict within 60s", canary)
case <-time.After(5 * time.Second):
t.Errorf("%s: no verdict within 5s", canary)
References
  1. Integration tests must be time bounded. Look for unbounded waits, time.Sleep longer than a few hundred milliseconds, or polling without a deadline. Prefer context.WithTimeout and t.Deadline(). (link)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowered to 30s in 17ed29d, the bound startExampleAgent and runExample already use for these SDK example processes. 5s is not enough for the first verdict: it includes starting a Node or Python process, enrolling, joining through the router and the first A2A call, with two language pairs running in parallel on a CI runner. The steady-state run of the whole test is about 5s; the bound is what stops a hang, and it fires well inside the package deadline.

ReverseProxy removes Forwarded and X-Forwarded-* before its Rewrite
runs and blanks a missing User-Agent, in Rewrite mode as in Director
mode; the test holds the proxy to it with a backend that records what it
received.

The canary script test waits 30s for a verdict, as the other SDK example
helpers do, instead of 60s.
@aojea
aojea merged commit b7a766f into google:main Sep 26, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant