sdk/js: reach peers through a router the control plane names by /dnsaddr - #513
Conversation
The testnets advertise their routers as /dnsaddr/bootstrap.<env>.sam-mesh.dev/p2p/<id>. The JS SDK built relay addresses on the address the credential carried, and js-libp2p resolves /dnsaddr/<host>/p2p/<router>/p2p-circuit/p2p/<peer> to the TXT records themselves, dropping the circuit, so every dial of a peer by ID failed with NoValidAddressesError. The js-calls-python-agent canary reported this on deploys 403 and 404 as the last 200 bytes of a stack trace. The Python SDK resolves the router address itself and dials the relay by peer ID, so python-calls-js-agent passed. An admitted router is now known by the address its connection was made on, resolved. A host takes a DNS resolver, and session.test.ts joins through a router handed out as /dnsaddr, with a stub resolver answering the TXT lookup, and reaches a peer through it; without the change the test fails with NoValidAddressesError. The canary script reports a failed call by the line that names the error, which Node prints before the stack trace and Python after, and takes its directory from SAM_CANARY_DIR. TestSDKCanaryScript runs the script from the template as the pod does: each language's agent example with its output in the shared directory, the other language's A2A example as the caller, both enrolled with one token, the second call resuming the identity, and a failing call reported by its error line.
x/tools v0.40.0's SSA builder panics on this tree under Go 1.27
("unexpected expr: *ast.KeyValueExpr"), so the deadcode step never ran.
golangci-lint v2.14.0 flags what v2.11.4 let through: ReverseProxy's
Director is deprecated since Go 1.26 and reflect.Ptr is an alias.
The egress proxy and the inference proxy use Rewrite. The inference
proxy's Rewrite is empty: ReverseProxy strips the inbound Forwarded and
X-Forwarded-* headers before calling it and blanks a missing User-Agent
itself, which is what the Director and the transport did by hand.
There was a problem hiding this comment.
Code Review
This pull request introduces several updates, including migrating httputil.ReverseProxy from Director to Rewrite in inference_service.go and sidecar.go, upgrading linter versions in hack/lint.sh, adding support for /dnsaddr router resolution in the JS SDK, and introducing integration tests for the SDK canary script. Feedback on these changes highlights two main issues: first, the empty Rewrite function in inference_service.go fails to strip Forwarded and X-Forwarded-* headers or blank missing User-Agent headers, which requires explicit handling; second, the integration test in sdk_canary_test.go violates the repository's style guide by using an excessively long 60-second timeout instead of being tightly time-bounded.
| // The transport addresses the backend; the proxy itself strips the | ||
| // inbound Forwarded and X-Forwarded-* headers and blanks a missing | ||
| // User-Agent. | ||
| Rewrite: func(*httputil.ProxyRequest) {}, |
There was a problem hiding this comment.
5. API surfaces and secrets / General Correctness & Security
The empty Rewrite function does not automatically strip Forwarded or X-Forwarded-* headers because ProxyRequest.SetURL is not called here. Consequently, any client-supplied X-Forwarded-* headers will be forwarded to the backend LLM/inference service since the explicit Del calls were removed from RoundTrip.
Additionally, the previous Director implementation blanked a missing User-Agent to prevent Go's default User-Agent (e.g., Go-http-client/1.1) from being appended. This blanking behavior is now lost.
Please update the Rewrite function to explicitly delete these headers and blank the User-Agent if it is missing.
// The transport addresses the backend; the proxy itself strips the
// inbound Forwarded and X-Forwarded-* headers and blanks a missing
// User-Agent.
Rewrite: func(pr *httputil.ProxyRequest) {
pr.Out.Header.Del("Forwarded")
pr.Out.Header.Del("X-Forwarded-For")
pr.Out.Header.Del("X-Forwarded-Host")
pr.Out.Header.Del("X-Forwarded-Proto")
if _, ok := pr.In.Header["User-Agent"]; !ok {
pr.Out.Header.Set("User-Agent", "")
}
},There was a problem hiding this comment.
This is not what ReverseProxy does. In ServeHTTP (Go 1.27, net/http/httputil/reverseproxy.go), when Rewrite is set the proxy deletes Forwarded, X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto from the outbound request before calling Rewrite, unconditionally; SetURL and SetXForwarded are what a Rewrite calls to add values back, and this one calls neither. The User-Agent blanking (if _, ok := outreq.Header["User-Agent"]; !ok { outreq.Header.Set("User-Agent", "") }) runs after both branches, so it applies in Rewrite mode as it did in Director mode; the Director's copy of it was redundant even before this change.
17ed29d adds TestInferenceService_ForwardingHeadersDoNotReachTheBackend, which sends the spoofed Forwarded/X-Forwarded-* headers and no User-Agent through svc.Handler() to a recording backend and asserts none of them arrive and the User-Agent is empty. With a Rewrite that copies the inbound headers through, the test fails on all four; with the empty Rewrite it passes.
| case <-time.After(60 * time.Second): | ||
| t.Errorf("%s: no verdict within 60s", canary) |
There was a problem hiding this comment.
3. Test coverage must follow the pyramid
Under Rule 3, integration tests must be time-bounded and should not have long or unbounded waits. The repository style guide specifies that integration tests are bounded to 10 seconds per execution.
Using a 60-second timeout via time.After(60 * time.Second) is excessively long for a test that is expected to run in ~5 seconds with INTERVAL=1. Please reduce this timeout to a much shorter duration (e.g., 5 seconds) to ensure the test fails quickly if it hangs, adhering to the 10-second execution limit.
| case <-time.After(60 * time.Second): | |
| t.Errorf("%s: no verdict within 60s", canary) | |
| case <-time.After(5 * time.Second): | |
| t.Errorf("%s: no verdict within 5s", canary) |
References
- Integration tests must be time bounded. Look for unbounded waits, time.Sleep longer than a few hundred milliseconds, or polling without a deadline. Prefer context.WithTimeout and t.Deadline(). (link)
There was a problem hiding this comment.
Lowered to 30s in 17ed29d, the bound startExampleAgent and runExample already use for these SDK example processes. 5s is not enough for the first verdict: it includes starting a Node or Python process, enrolling, joining through the router and the first A2A call, with two language pairs running in parallel on a CI runner. The steady-state run of the whole test is about 5s; the bound is what stops a hang, and it fires well inside the package deadline.
ReverseProxy removes Forwarded and X-Forwarded-* before its Rewrite runs and blanks a missing User-Agent, in Rewrite mode as in Director mode; the test holds the proxy to it with a backend that records what it received. The canary script test waits 30s for a verdict, as the other SDK example helpers do, instead of 60s.
The
js-calls-python-agentcanary failed on deploys #403 and #404 with the last 200 bytes of a stack trace. This is the cause, the fix, and the tests that would have caught it before a deploy.Cause
The testnets advertise their routers as
/dnsaddr/bootstrap.<env>.sam-mesh.dev/p2p/<id>(router--external-addr). The JS SDK built relay addresses on the address the credential carried, so a call by peer ID dialed/dnsaddr/<host>/p2p/<router>/p2p-circuit/p2p/<peer>. js-libp2p 3.3.11 resolves such an address to the TXT records themselves and drops what follows the dnsaddr's/p2p/<router>; the dial queue then discards the records as the wrong peer and throwsNoValidAddressesError. Every JS call by peer ID failed in the testnet. The Python SDK resolves the router address itself and dials the relay by peer ID, sopython-calls-js-agentpassed.The integration harness's router advertises
/ip4addresses, so nothing exercised/dnsaddr; the images built from the same commits run the pair fine locally.Fix
An admitted router is known by the resolved address its connection was made on (
conn.remoteAddr), so a relay address never carries a/dnsaddr. A mesh host takes adnsresolver so the case is testable.Tests
sdk/js/src/session.test.ts: joins through a router handed out as/dnsaddr/router.test/p2p/<id>, with a stub resolver answering the TXT lookup, and authenticates a peer through it. Without the fix it fails withNoValidAddressesError.tests/integration/sdk_canary_test.go(TestSDKCanaryScript, ~5s): runscanary.shfrom.github/k8s/sam-sdk-canary-template.yamlas the pod does — each language's agent example logging to the shared directory, the other language's A2A example asCALL, both enrolled with one token, the second call resuming the identity — and checks the verdict lines and the readiness file. It also feeds canned Node and Python failures and checks the error line reported.SAM_CANARY_DIR.Lint
A second commit makes
hack/lint.shpass: golangci-lint v2.14.0 and deadcode from x/tools v0.50.0 (v0.40.0's SSA builder panics under Go 1.27, so the deadcode step never ran), the two remainingReverseProxy.Directoruses moved toRewrite,reflect.Ptr→reflect.Pointer, and ago fmtfix.Validation:
npm run typecheck,npm test(66),TestNativeSDKsMesh|TestNativeSDKA2A|TestNativeSDKExamples|TestSDKCanaryScript,go build ./...,go test ./internal/node ./internal/storage ./internal/standalone,hack/lint.shexit 0.