Skip to content

Engine body limit + req host, photos and graph media planes for migration e2e - #3

Merged
deblasis merged 5 commits into
mainfrom
feat/migration-e2e-support
Aug 3, 2026
Merged

Engine body limit + req host, photos and graph media planes for migration e2e#3
deblasis merged 5 commits into
mainfrom
feat/migration-e2e-support

Conversation

@deblasis

@deblasis deblasis commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Workstream A of the cloud-migration slice: the stunt-side engine and adapter work that lets the onlin migration e2e run entirely against local sims.

Engine

  • Configurable request body limit, honest overflow. New per-service max_body_bytes in stunt.yaml (default stays 1 MiB). The MaxBytesReader error is no longer discarded: overflow returns 413, other read failures 400, and a handler never sees truncated bytes.
  • Found while fixing the above: the request-log recorder was pre-reading the body through a 64 KB LimitReader and replacing req.Body with just those bytes, so served requests over 64 KB were already truncated before the 1 MiB cap. The recorder now tees the stream: capture stays capped at 64 KB for the log, the handler reads the full body. This was not in the design (it cited only engine.go:447) but the feature is unshippable without it.
  • Host visibility. req["host"] (Go's r.Host) is injected into the Starlark request dict so handlers can mint self-referential URLs. Reference docs (AGENTS.md, stunt llm) updated.

photos-style

  • Uploads store the raw request bytes in the blob store keyed by the generated uploadToken (generated ids only, valid blob charset), recording the request Content-Type; batchCreate links the blob to the created media item.
  • baseUrl is computed at read time from req["host"] as http://{host}/v1/media-dl/{id}; no host is stored in documents.
  • New GET /v1/media-dl/{id} with strict Google suffix semantics: =d/=dv (arriving inside the captured param) serve the original bytes with the recorded content type; a bare baseUrl serves a deterministic derivative payload with different bytes so a client that forgets =d fails byte-comparison loudly. Unknown ids 404.
  • New GET /v1/mediaItems/{id}; list and search honor pageSize/pageToken and emit nextPageToken.

microsoft-graph-style (OneDrive write plane, strict)

  • Colon addressing via fixed-depth routes with the colon inside the captured segment ({param}: is not routable); handlers strip and require the trailing colon, 400 on malformed addressing.
  • Simple upload PUT root:/{name}:/content + items/{parentId}:/{name}:/content: 201 driveItem (id, name, size, parentReference); repeat PUT replaces in place (200, same id); @microsoft.graph.conflictBehavior=rename produces name (1).ext; fail returns 409.
  • POST .../createUploadSession (root + folder variants) returns {uploadUrl: http://{host}/v1.0/_upload/{session}, expirationDateTime}.
  • PUT /v1.0/_upload/{session}: strict resumable protocol. Content-Range: bytes {start}-{end}/{total} parsed and enforced: start must equal the next expected offset, end >= start, end < total, total consistent across chunks, body length matching the declared range; violations 416, malformed header 400. Mid-session 202 + nextExpectedRanges; final range assembles the blob, creates the driveItem, 201; session deleted so later chunks 404.
  • GET items/{id}/content serves stored bytes verbatim; createFolder via POST root/children and items/{id}/children (default fail 409, rename honored); GET root:/{path}:/?select=id path resolution; child listing is per-parent (files docs gained parentId).
  • GET /v1.0/me/drive already satisfies a select=quota query with its quota object, verified by test and left as is.

Tests

10 new Go engine-level tests (plus subtests), all booting the real adapters over real HTTP, TDD-first:

  • body limit: default oversize 413, just-under-limit byte-exact round-trip, small limit rejects, raised limit accepts 3 MiB
  • req["host"] echo matches the listen address
  • photos: upload -> batchCreate -> get -> media-dl =d/=dv byte-equality, derivative discrimination, 404s, multi-page list and search pagination
  • graph: simple upload + replace + rename + fail + 401/400, folders/resolution/per-parent listing/quota, full chunked session with the 416 matrix and post-completion rejection, folder session variant, 413 with small max_body_bytes

Byte-fidelity fixtures contain all 256 byte values behind a PNG magic prefix, so any future sanitization breaks loudly. Binary planes are covered here because stunt adapter test traces are JSON-only.

just ci green (build, race tests, vet, gofmt, mod tidy, cross-build, lint of all 91 adapters). DISCLAIMER files untouched; stunt adapter lint clean for both adapters.

Deviations from the design

  • The recorder-level 64 KB truncation fix (above) was necessary but unanticipated.
  • Simple upload default conflict behavior is replace-in-place (200, same id) matching real Graph PUT semantics; the design only named the rename case.
  • Session creation returns 200 (real Graph) rather than 201; completed sessions answer 404 (session deleted), which is how the real service behaves for invalidated upload URLs.

Two engine changes needed for self-hosted media planes in adapters.

Body limit: request bodies over the limit used to be silently truncated
twice: the request-log recorder replaced the body stream with its first
64 KB, and the dispatch path discarded the MaxBytesReader error at
1 MiB. Handlers could receive partial data and never know. Services can
now set max_body_bytes in stunt.yaml (default stays 1 MiB). The read
error is propagated: overflow returns 413, other read failures return
400, and a handler never sees truncated bytes. The recorder now tees
the body instead of pre-reading it, so capture stays capped at 64 KB
for the log while the full stream reaches the handler untouched.

Host visibility: Go moves the Host header to r.Host, so Starlark
handlers could never see the address a client used and could not mint
self-referential URLs (photos baseUrl, upload session uploadUrl). The
request dict now carries req["host"] = r.Host.

Covered by engine tests: default oversize 413, just-under-limit body
round-trips byte-exact (all 256 byte values incl. PNG magic), small
limit rejects, raised limit accepts 3 MiB, handler echoes req["host"]
matching the listen address. Reference docs (AGENTS.md, stunt llm)
updated.
The upload pipeline previously minted tokens but threw the bytes away,
and baseUrl pointed at a fake host baked in at create time. Migration
e2e needs the sim to actually hold and serve media bytes.

Uploads now store the raw request body in the blob store keyed by the
generated uploadToken, with the request Content-Type recorded.
batchCreate copies the blob to the created media item id. baseUrl is
computed at read time from req["host"] as
http://{host}/v1/media-dl/{id}; no host is stored in documents.

New GET /v1/media-dl/{id} serves the bytes. The {id} segment may carry
the Google download suffix (=d or =dv) inside the captured value. Only
the suffixed forms return the original bytes (with the recorded content
type); a bare baseUrl returns a deterministic derivative payload with
clearly different bytes, so a client that forgets =d fails
byte-comparison loudly instead of silently passing. Unknown ids 404.

New GET /v1/mediaItems/{id} returns the public item. List and search
now honor pageSize and pageToken (offset cursor) and emit nextPageToken
while more items remain.

Engine tests cover the full flow: upload of an all-256-byte-values
fixture, batchCreate link, read-time baseUrl, =d and =dv byte-equality,
derivative discrimination, 404s, and multi-page list and search
pagination. Adapter lint stays clean.
Adds the OneDrive write surface with real Graph shapes, implemented
strictly so client protocol bugs cannot hide behind a lenient mock.

Simple upload: PUT /v1.0/me/drive/root:/{name}:/content and the
items/{parentId}:/{name}:/content variant. The router cannot treat
'{param}:' as a parameter, so routes use fixed-depth segments with the
colon inside the captured value; handlers strip and require the
trailing colon (400 on malformed addressing). Creates return 201 with
the driveItem (id, name, size, parentReference); a repeat PUT replaces
in place (200, same id); conflictBehavior=rename creates 'name (1).ext'
siblings; fail returns 409.

Resumable upload: POST createUploadSession (root and folder variants)
stores a session and returns an uploadUrl minted from req["host"]
(http://{host}/v1.0/_upload/{session}). PUT chunks parse Content-Range
'bytes {start}-{end}/{total}' and enforce the real protocol: start must
equal the next expected offset, end >= start, end < total, totals
consistent across chunks, body length matching the declared range; any
violation is 416, a malformed header is 400. Mid-session chunks answer
202 with expirationDateTime and nextExpectedRanges; the final range
assembles the blob, creates the driveItem, returns 201, and deletes the
session so later chunks get 404.

Also: GET items/{id}/content serves stored bytes verbatim, createFolder
via POST root/children and items/{id}/children (default fail with 409,
rename honored), GET root:/{path}:/ resolves a path with ?select=id
support, and child listing is per-parent (files docs gained parentId;
seeds updated). GET /v1.0/me/drive already satisfies select=quota and
is unchanged.

Engine tests cover the full happy paths end to end over real HTTP
(all-256-byte-values fixtures, byte-equality on download), the 416
matrix, session invalidation, folder scoping, rename conflicts, and a
413 on oversize bodies with a small max_body_bytes. Adapter lint stays
clean.
… determinism

Review follow-ups on the migration e2e branch: both adapters exist to test
uploads, so the engine's default 1 MiB body cap deserves a loud note with a
ready-to-paste max_body_bytes example. The graph README also documents why
upload session ids are deterministic counters rather than unguessable
tokens, and warns against exposing a stunt server on a shared network.
@deblasis
deblasis marked this pull request as ready for review August 3, 2026 03:22
@deblasis
deblasis merged commit 821a8a4 into main Aug 3, 2026
1 check passed
@deblasis
deblasis deleted the feat/migration-e2e-support branch August 3, 2026 03:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant