Engine body limit + req host, photos and graph media planes for migration e2e - #3
Merged
Conversation
Two engine changes needed for self-hosted media planes in adapters. Body limit: request bodies over the limit used to be silently truncated twice: the request-log recorder replaced the body stream with its first 64 KB, and the dispatch path discarded the MaxBytesReader error at 1 MiB. Handlers could receive partial data and never know. Services can now set max_body_bytes in stunt.yaml (default stays 1 MiB). The read error is propagated: overflow returns 413, other read failures return 400, and a handler never sees truncated bytes. The recorder now tees the body instead of pre-reading it, so capture stays capped at 64 KB for the log while the full stream reaches the handler untouched. Host visibility: Go moves the Host header to r.Host, so Starlark handlers could never see the address a client used and could not mint self-referential URLs (photos baseUrl, upload session uploadUrl). The request dict now carries req["host"] = r.Host. Covered by engine tests: default oversize 413, just-under-limit body round-trips byte-exact (all 256 byte values incl. PNG magic), small limit rejects, raised limit accepts 3 MiB, handler echoes req["host"] matching the listen address. Reference docs (AGENTS.md, stunt llm) updated.
The upload pipeline previously minted tokens but threw the bytes away,
and baseUrl pointed at a fake host baked in at create time. Migration
e2e needs the sim to actually hold and serve media bytes.
Uploads now store the raw request body in the blob store keyed by the
generated uploadToken, with the request Content-Type recorded.
batchCreate copies the blob to the created media item id. baseUrl is
computed at read time from req["host"] as
http://{host}/v1/media-dl/{id}; no host is stored in documents.
New GET /v1/media-dl/{id} serves the bytes. The {id} segment may carry
the Google download suffix (=d or =dv) inside the captured value. Only
the suffixed forms return the original bytes (with the recorded content
type); a bare baseUrl returns a deterministic derivative payload with
clearly different bytes, so a client that forgets =d fails
byte-comparison loudly instead of silently passing. Unknown ids 404.
New GET /v1/mediaItems/{id} returns the public item. List and search
now honor pageSize and pageToken (offset cursor) and emit nextPageToken
while more items remain.
Engine tests cover the full flow: upload of an all-256-byte-values
fixture, batchCreate link, read-time baseUrl, =d and =dv byte-equality,
derivative discrimination, 404s, and multi-page list and search
pagination. Adapter lint stays clean.
Adds the OneDrive write surface with real Graph shapes, implemented
strictly so client protocol bugs cannot hide behind a lenient mock.
Simple upload: PUT /v1.0/me/drive/root:/{name}:/content and the
items/{parentId}:/{name}:/content variant. The router cannot treat
'{param}:' as a parameter, so routes use fixed-depth segments with the
colon inside the captured value; handlers strip and require the
trailing colon (400 on malformed addressing). Creates return 201 with
the driveItem (id, name, size, parentReference); a repeat PUT replaces
in place (200, same id); conflictBehavior=rename creates 'name (1).ext'
siblings; fail returns 409.
Resumable upload: POST createUploadSession (root and folder variants)
stores a session and returns an uploadUrl minted from req["host"]
(http://{host}/v1.0/_upload/{session}). PUT chunks parse Content-Range
'bytes {start}-{end}/{total}' and enforce the real protocol: start must
equal the next expected offset, end >= start, end < total, totals
consistent across chunks, body length matching the declared range; any
violation is 416, a malformed header is 400. Mid-session chunks answer
202 with expirationDateTime and nextExpectedRanges; the final range
assembles the blob, creates the driveItem, returns 201, and deletes the
session so later chunks get 404.
Also: GET items/{id}/content serves stored bytes verbatim, createFolder
via POST root/children and items/{id}/children (default fail with 409,
rename honored), GET root:/{path}:/ resolves a path with ?select=id
support, and child listing is per-parent (files docs gained parentId;
seeds updated). GET /v1.0/me/drive already satisfies select=quota and
is unchanged.
Engine tests cover the full happy paths end to end over real HTTP
(all-256-byte-values fixtures, byte-equality on download), the 416
matrix, session invalidation, folder scoping, rename conflicts, and a
413 on oversize bodies with a small max_body_bytes. Adapter lint stays
clean.
… determinism Review follow-ups on the migration e2e branch: both adapters exist to test uploads, so the engine's default 1 MiB body cap deserves a loud note with a ready-to-paste max_body_bytes example. The graph README also documents why upload session ids are deterministic counters rather than unguessable tokens, and warns against exposing a stunt server on a shared network.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workstream A of the cloud-migration slice: the stunt-side engine and adapter work that lets the onlin migration e2e run entirely against local sims.
Engine
max_body_bytesinstunt.yaml(default stays 1 MiB). TheMaxBytesReadererror is no longer discarded: overflow returns 413, other read failures 400, and a handler never sees truncated bytes.LimitReaderand replacingreq.Bodywith just those bytes, so served requests over 64 KB were already truncated before the 1 MiB cap. The recorder now tees the stream: capture stays capped at 64 KB for the log, the handler reads the full body. This was not in the design (it cited only engine.go:447) but the feature is unshippable without it.req["host"](Go'sr.Host) is injected into the Starlark request dict so handlers can mint self-referential URLs. Reference docs (AGENTS.md,stunt llm) updated.photos-style
baseUrlis computed at read time fromreq["host"]ashttp://{host}/v1/media-dl/{id}; no host is stored in documents.GET /v1/media-dl/{id}with strict Google suffix semantics:=d/=dv(arriving inside the captured param) serve the original bytes with the recorded content type; a bare baseUrl serves a deterministic derivative payload with different bytes so a client that forgets=dfails byte-comparison loudly. Unknown ids 404.GET /v1/mediaItems/{id}; list and search honorpageSize/pageTokenand emitnextPageToken.microsoft-graph-style (OneDrive write plane, strict)
{param}:is not routable); handlers strip and require the trailing colon, 400 on malformed addressing.PUT root:/{name}:/content+items/{parentId}:/{name}:/content: 201 driveItem (id, name, size, parentReference); repeat PUT replaces in place (200, same id);@microsoft.graph.conflictBehavior=renameproducesname (1).ext;failreturns 409.POST .../createUploadSession(root + folder variants) returns{uploadUrl: http://{host}/v1.0/_upload/{session}, expirationDateTime}.PUT /v1.0/_upload/{session}: strict resumable protocol.Content-Range: bytes {start}-{end}/{total}parsed and enforced: start must equal the next expected offset, end >= start, end < total, total consistent across chunks, body length matching the declared range; violations 416, malformed header 400. Mid-session 202 +nextExpectedRanges; final range assembles the blob, creates the driveItem, 201; session deleted so later chunks 404.GET items/{id}/contentserves stored bytes verbatim; createFolder viaPOST root/childrenanditems/{id}/children(default fail 409, rename honored);GET root:/{path}:/?select=idpath resolution; child listing is per-parent (files docs gainedparentId).GET /v1.0/me/drivealready satisfies aselect=quotaquery with its quota object, verified by test and left as is.Tests
10 new Go engine-level tests (plus subtests), all booting the real adapters over real HTTP, TDD-first:
req["host"]echo matches the listen address=d/=dvbyte-equality, derivative discrimination, 404s, multi-page list and search paginationmax_body_bytesByte-fidelity fixtures contain all 256 byte values behind a PNG magic prefix, so any future sanitization breaks loudly. Binary planes are covered here because
stunt adapter testtraces are JSON-only.just cigreen (build, race tests, vet, gofmt, mod tidy, cross-build, lint of all 91 adapters). DISCLAIMER files untouched;stunt adapter lintclean for both adapters.Deviations from the design