diff --git a/AGENTS.md b/AGENTS.md index 6037754..0b53d74 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,7 @@ services: # respond: { status: 200, body: { inline: { message: hi } } } config: # optional per-service config passed to adapter scripts webhook_url: http://127.0.0.1:9999/hooks + max_body_bytes: 8388608 # optional request-body cap (default 1 MiB); oversize → 413 ``` ### Rule fields (inline declarative responses) @@ -162,6 +163,7 @@ must **return a response** via `respond(...)` (or a dict shaped `{status, body, |---|---|---| | `req["method"]` | string | HTTP method (`GET`, `POST`, ...) | | `req["path"]` | string | request path | +| `req["host"]` | string | request Host header (`127.0.0.1:8000`); use it to mint self-referential URLs | | `req["headers"]` | dict | request headers (case-insensitive keys); e.g. `req["headers"]["Authorization"]` | | `req["body"]` | dict \| list \| None | parsed JSON body (None if no/invalid JSON) | | `req["raw_body"]` | string | raw body bytes as a string (for non-JSON/binary uploads) | diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e55a03..ec8866e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,34 @@ All notable changes to **stunt** are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Engine + +- **Configurable request body limit with honest overflow.** Services can set + `max_body_bytes` in `stunt.yaml` (default stays 1 MiB). Oversize bodies now + return **413** instead of being silently truncated; the request-log recorder + tees the body stream (capture stays capped at 64 KB) so handlers always see + the full, untruncated bytes. +- **`req["host"]` in Starlark handlers.** The request Host header is injected + into the request dict, so adapters can mint self-referential URLs (media + `baseUrl`, upload session `uploadUrl`) that point back at the simulator. + +### Adapters + +- **photos-style:** real media plane. Uploaded bytes are stored and linked to + created media items; `baseUrl` is computed at read time from the request + host; new `GET /v1/media-dl/{id}` with strict `=d`/`=dv` semantics (bare + baseUrl serves a distinct derivative payload); new `GET /v1/mediaItems/{id}`; + list/search honor `pageSize`/`pageToken` and emit `nextPageToken`. +- **microsoft-graph-style:** strict OneDrive write plane. Simple upload + (`PUT root:/{name}:/content` + folder variant) with real conflictBehavior + semantics, createFolder, per-parent child listing, path resolution with + `?select=id`, `GET items/{id}/content`, and the full resumable upload + protocol (`createUploadSession`, self-referential `uploadUrl`, sequential + Content-Range chunks with 416 on violations, 202 + `nextExpectedRanges`, + 201 + driveItem on the final range, session invalidation). + ## [0.2.2] — 2026-07-24 ### Housekeeping diff --git a/adapters/microsoft-graph-style/README.md b/adapters/microsoft-graph-style/README.md index d47a5be..08a4b6e 100644 --- a/adapters/microsoft-graph-style/README.md +++ b/adapters/microsoft-graph-style/README.md @@ -19,7 +19,26 @@ real API data is included. - **Outlook mail:** `GET /v1.0/me/messages`, `GET /v1.0/me/messages/{id}`, `POST /v1.0/me/sendMail` (202, STATEFUL), `GET /v1.0/me/mailFolders`. - **Calendar:** `GET /v1.0/me/events`, `POST /v1.0/me/events` (STATEFUL). -- **OneDrive:** `GET /v1.0/me/drive`, `GET /v1.0/me/drive/root/children`. +- **OneDrive (read):** `GET /v1.0/me/drive` (incl. quota), + `GET /v1.0/me/drive/root/children`, `GET /v1.0/me/drive/items/{id}/children` + (listing is per-parent), `GET /v1.0/me/drive/items/{id}/content` (stored + bytes verbatim), `GET /v1.0/me/drive/root:/{path}:/` path resolution + (supports `?select=id`). +- **OneDrive (write):** real Graph colon addressing, implemented strictly. + Simple upload `PUT /v1.0/me/drive/root:/{name}:/content` (and the + `items/{parentId}:/{name}:/content` variant) → 201 driveItem; a repeat PUT + replaces (200), `@microsoft.graph.conflictBehavior=rename` creates + `name (1).ext` siblings, `fail` returns 409. createFolder via + `POST .../root/children` and `POST .../items/{id}/children`. +- **OneDrive resumable uploads:** `POST .../createUploadSession` returns a + self-referential `uploadUrl` (`http://{host}/v1.0/_upload/{session}`); + `PUT` chunks must carry `Content-Range: bytes {start}-{end}/{total}` and be + sequential and contiguous — wrong offset, gaps, inconsistent totals, or + range/body mismatches return **416**; mid-session chunks return 202 with + `nextExpectedRanges`; the final range assembles the file and returns 201 + with the driveItem; the session is invalidated afterwards (further chunks + 404). Strictness is deliberate: a lenient mock would mask client protocol + bugs. - **SharePoint:** `GET /v1.0/groups/{id}/sites`. - **Teams chats:** `GET /v1.0/me/chats`, `POST /v1.0/me/chats`, `GET /v1.0/chats/{id}/messages`, `POST /v1.0/chats/{id}/messages` (STATEFUL). @@ -45,8 +64,18 @@ endpoints, with `@odata.nextLink` pagination. | POST | `/v1.0/me/sendMail` | `mail.star#on_send_mail` | Send mail → 202 | | GET | `/v1.0/me/events` | `calendar.star#on_list_events` | List events (OData) | | POST | `/v1.0/me/events` | `calendar.star#on_create_event` | Create event | -| GET | `/v1.0/me/drive` | `drive.star#on_get_drive` | Drive info | +| GET | `/v1.0/me/drive` | `drive.star#on_get_drive` | Drive info (incl. quota) | | GET | `/v1.0/me/drive/root/children` | `drive.star#on_list_children` | Root children | +| POST | `/v1.0/me/drive/root/children` | `drive.star#on_create_child_root` | createFolder (root) | +| GET | `/v1.0/me/drive/items/{id}/children` | `drive.star#on_list_item_children` | Folder children | +| POST | `/v1.0/me/drive/items/{id}/children` | `drive.star#on_create_child_item` | createFolder (nested) | +| GET | `/v1.0/me/drive/items/{id}/content` | `drive_upload.star#on_get_content` | Download stored bytes | +| PUT | `/v1.0/me/drive/root:/{name}:/content` | `drive_upload.star#on_simple_upload_root` | Simple upload (root) | +| PUT | `/v1.0/me/drive/items/{parentId}:/{name}:/content` | `drive_upload.star#on_simple_upload_item` | Simple upload (folder) | +| POST | `/v1.0/me/drive/root:/{name}:/createUploadSession` | `drive_upload.star#on_create_session_root` | Resumable session (root) | +| POST | `/v1.0/me/drive/items/{parentId}:/{name}:/createUploadSession` | `drive_upload.star#on_create_session_item` | Resumable session (folder) | +| PUT | `/v1.0/_upload/{session}` | `drive_upload.star#on_upload_chunk` | Strict chunk PUT (416 on violations) | +| GET | `/v1.0/me/drive/root:/{path}:/` | `drive_upload.star#on_resolve_path` | Path resolution (`?select=id`) | | GET | `/v1.0/groups/{id}/sites` | `sharepoint.star#on_list_sites` | SharePoint sites | | GET | `/v1.0/me/chats` | `teams.star#on_list_chats` | List chats (OData) | | POST | `/v1.0/me/chats` | `teams.star#on_create_chat` | Create chat | @@ -63,13 +92,19 @@ endpoints, with `@odata.nextLink` pagination. | `events` | Calendar events | | `chats` | Teams chats | | `chat_messages` | Teams chat messages (per chat) | -| `files` | OneDrive files/folders | +| `files` | OneDrive files/folders (with `parentId` for per-parent listing) | +| `sessions` | OneDrive resumable upload sessions (next offset, total, conflict behavior) | + +File content lives in the blob store, keyed by driveItem id (in-flight +session chunks accumulate under `up-{session}` until the final range). ## Auth -All endpoints require `Authorization: Bearer `. The token value is not validated — -only presence is checked. A missing header returns `401` with a Graph error envelope -(`{error:{code, message}}`). +All endpoints require `Authorization: Bearer `, except +`PUT /v1.0/_upload/{session}` — real upload session URLs are +pre-authenticated, so the sim matches. The token value is not validated — +only presence is checked. A missing header returns `401` with a Graph error +envelope (`{error:{code, message}}`). ## Usage @@ -77,6 +112,18 @@ only presence is checked. A missing header returns `401` with a Graph error enve services: graph: adapter: ./adapters/microsoft-graph-style + max_body_bytes: 33554432 # uploads over 1 MiB need a raised body limit ``` Then `stunt up` and make requests to the served address. + +Note: the engine's default request-body limit is 1 MiB and oversize bodies +get a `413`; set `max_body_bytes` (as above) when testing uploads or chunk +PUTs over 1 MiB. + +Upload session URLs (`/v1.0/_upload/sess-NNNNNN`) carry no bearer check, +matching real Graph where the upload URL is pre-authenticated. Unlike real +Graph the session ids here are deterministic monotonic counters, not +unguessable tokens: stunt ids are deterministic by design (`rng_seed` +reproducibility) and the sim binds to localhost. Do not expose a stunt +server to a shared network. diff --git a/adapters/microsoft-graph-style/adapter.yaml b/adapters/microsoft-graph-style/adapter.yaml index bc5bccb..0598344 100644 --- a/adapters/microsoft-graph-style/adapter.yaml +++ b/adapters/microsoft-graph-style/adapter.yaml @@ -57,12 +57,52 @@ endpoints: handler: scripts/calendar.star#on_create_event # --- OneDrive --- + # NOTE on colon addressing: the router does not recognize "{param}:" as a + # parameter, but a literal ':' inside a captured VALUE is fine. Routes below + # use fixed-depth segments where the colon lands inside the captured + # segment: /root:/{item}/content matches /root:/photo.jpg:/content with + # item="photo.jpg:" and the handler strips the trailing colon. - route: /v1.0/me/drive method: GET handler: scripts/drive.star#on_get_drive - route: /v1.0/me/drive/root/children method: GET handler: scripts/drive.star#on_list_children + - route: /v1.0/me/drive/root/children + method: POST + handler: scripts/drive.star#on_create_child_root + - route: /v1.0/me/drive/items/{id}/children + method: GET + handler: scripts/drive.star#on_list_item_children + - route: /v1.0/me/drive/items/{id}/children + method: POST + handler: scripts/drive.star#on_create_child_item + - route: /v1.0/me/drive/items/{id}/content + method: GET + handler: scripts/drive_upload.star#on_get_content + # Simple upload (< 4 MB path): PUT root:/{name}:/content and the + # items/{parentId}:/{name}:/content variant. + - route: /v1.0/me/drive/root:/{item}/content + method: PUT + handler: scripts/drive_upload.star#on_simple_upload_root + - route: /v1.0/me/drive/items/{parent}/{item}/content + method: PUT + handler: scripts/drive_upload.star#on_simple_upload_item + # Resumable upload sessions. + - route: /v1.0/me/drive/root:/{item}/createUploadSession + method: POST + handler: scripts/drive_upload.star#on_create_session_root + - route: /v1.0/me/drive/items/{parent}/{item}/createUploadSession + method: POST + handler: scripts/drive_upload.star#on_create_session_item + - route: /v1.0/_upload/{session} + method: PUT + handler: scripts/drive_upload.star#on_upload_chunk + # Path resolution (GET root:/{path}:/ with ?select=id) — parameterized, + # declared after the literal root:/... routes above. + - route: /v1.0/me/drive/root:/{item} + method: GET + handler: scripts/drive_upload.star#on_resolve_path # --- SharePoint --- - route: /v1.0/groups/{id}/sites @@ -103,6 +143,8 @@ resources: kind: collection - name: files kind: collection + - name: sessions + kind: collection # Auth scheme metadata (mock: any Bearer token accepted; presence checked). identity: diff --git a/adapters/microsoft-graph-style/scripts/drive.star b/adapters/microsoft-graph-style/scripts/drive.star index e49400e..4d17432 100644 --- a/adapters/microsoft-graph-style/scripts/drive.star +++ b/adapters/microsoft-graph-style/scripts/drive.star @@ -1,9 +1,18 @@ -# Microsoft Graph v1.0 — OneDrive handlers. +# Microsoft Graph v1.0 — OneDrive metadata handlers. # -# GET /v1.0/me/drive → default drive info -# GET /v1.0/me/drive/root/children → root folder children (files/folders) +# GET /v1.0/me/drive → default drive info (incl. quota) +# GET /v1.0/me/drive/root/children → root folder children +# POST /v1.0/me/drive/root/children → createFolder under root +# GET /v1.0/me/drive/items/{id}/children → children of a folder +# POST /v1.0/me/drive/items/{id}/children → createFolder inside a folder +# +# Listing is PER-PARENT: every files doc carries a parentId ("root" for the +# drive root) and children endpoints filter by it. The upload plane lives in +# drive_upload.star; shared driveItem helpers live in lib.star. -# on_get_drive returns the default drive for the current user. +# on_get_drive returns the default drive for the current user. The response +# always carries the quota object, so a ?select=quota (or $select=quota) +# query is satisfied by the same shape. # GET /v1.0/me/drive (Bearer) def on_get_drive(req): err = _require_bearer(req) @@ -36,27 +45,95 @@ def on_list_children(req): return err _seed_files() + entities = _children_entities("root") + base_url = "https://graph.microsoft.com/v1.0/me/drive/root/children" + return _apply_odata(entities, req["query"], base_url) + +# on_list_item_children returns the children of a folder by id. +# GET /v1.0/me/drive/items/{id}/children (Bearer) +def on_list_item_children(req): + err = _require_bearer(req) + if err != None: + return err + + _seed_files() + parent_id = req["params"]["id"] fc = store_collection("files") - docs = fc.list() - entities = [] - for d in docs: - entities.append(_file_entity(d)) + if fc.get(parent_id) == None: + return _err("itemNotFound", 404, "The resource could not be found.") - base_url = "https://graph.microsoft.com/v1.0/me/drive/root/children" + entities = _children_entities(parent_id) + base_url = "https://graph.microsoft.com/v1.0/me/drive/items/" + parent_id + "/children" return _apply_odata(entities, req["query"], base_url) +# on_create_child_root creates a folder under the drive root. +# POST /v1.0/me/drive/root/children (Bearer; {name, folder: {}}) +def on_create_child_root(req): + err = _require_bearer(req) + if err != None: + return err + return _create_folder(req, "root") + +# on_create_child_item creates a folder inside an existing folder. +# POST /v1.0/me/drive/items/{id}/children (Bearer; {name, folder: {}}) +def on_create_child_item(req): + err = _require_bearer(req) + if err != None: + return err + + _seed_files() + parent_id = req["params"]["id"] + fc = store_collection("files") + parent = fc.get(parent_id) + if parent == None or parent.get("folder") == None: + return _err("itemNotFound", 404, "The parent folder could not be found.") + return _create_folder(req, parent_id) + # --- helpers --- -def _file_entity(doc): - return { - "id": doc["id"], - "name": doc["name"], - "file": doc.get("file", None), - "folder": doc.get("folder", None), - "size": doc.get("size", 0), - "createdDateTime": doc.get("createdDateTime", "2024-01-01T00:00:00Z"), - "lastModifiedDateTime": doc.get("lastModifiedDateTime", "2024-01-01T00:00:00Z"), +# _create_folder handles the createFolder body against a parent. Graph's +# default conflict behavior for createFolder is "fail" (409 +# nameAlreadyExists); "rename" appends " (1)"-style suffixes. +def _create_folder(req, parent_id): + body = req["body"] + if body == None: + body = {} + name = body.get("name", "") + if name == "" or body.get("folder") == None: + return _err("invalidRequest", 400, "A folder item requires 'name' and a 'folder' facet.") + + conflict = body.get("@microsoft.graph.conflictBehavior", "fail") + fc = store_collection("files") + existing = _find_child_by_name(fc, parent_id, name) + if existing != None: + if conflict == "rename": + name = _conflict_rename(fc, parent_id, name) + elif conflict == "replace": + return respond(200, _drive_item(existing)) + else: + return _err("nameAlreadyExists", 409, "An item with the same name already exists under the parent.") + + doc = { + "id": _next_item_id(), + "name": name, + "file": None, + "folder": {"childCount": 0}, + "size": 0, + "parentId": parent_id, + "createdDateTime": "2024-06-15T12:00:00Z", + "lastModifiedDateTime": "2024-06-15T12:00:00Z", } + fc.insert(doc) + return respond(201, _drive_item(doc)) + +# _children_entities lists the driveItems whose parentId matches. +def _children_entities(parent_id): + fc = store_collection("files") + entities = [] + for d in fc.list(): + if d.get("parentId", "root") == parent_id: + entities.append(_drive_item(d)) + return entities def _seed_files(): fc = store_collection("files") @@ -70,6 +147,7 @@ def _seed_files(): "file": {"mimeType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document"}, "folder": None, "size": 24576, + "parentId": "root", "createdDateTime": "2024-03-01T10:00:00Z", "lastModifiedDateTime": "2024-06-10T15:30:00Z", }, @@ -79,6 +157,7 @@ def _seed_files(): "file": {"mimeType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"}, "folder": None, "size": 53248, + "parentId": "root", "createdDateTime": "2024-02-15T09:00:00Z", "lastModifiedDateTime": "2024-06-12T11:00:00Z", }, @@ -88,6 +167,7 @@ def _seed_files(): "file": None, "folder": {"childCount": 5}, "size": 0, + "parentId": "root", "createdDateTime": "2024-01-20T08:00:00Z", "lastModifiedDateTime": "2024-06-14T16:00:00Z", }, diff --git a/adapters/microsoft-graph-style/scripts/drive_upload.star b/adapters/microsoft-graph-style/scripts/drive_upload.star new file mode 100644 index 0000000..5b948d0 --- /dev/null +++ b/adapters/microsoft-graph-style/scripts/drive_upload.star @@ -0,0 +1,319 @@ +# Microsoft Graph v1.0 — OneDrive upload plane (simple + resumable) and +# content download. Implements the REAL Graph shapes STRICTLY: a lenient sim +# would let client protocol bugs (wrong offsets, parallel chunks, misparsed +# 202 bodies) hide behind the mock. +# +# PUT /v1.0/me/drive/root:/{name}:/content → simple upload (root) +# PUT /v1.0/me/drive/items/{parentId}:/{name}:/content → simple upload (folder) +# GET /v1.0/me/drive/root:/{path}:/ → path resolution (?select=id) +# POST /v1.0/me/drive/root:/{name}:/createUploadSession → resumable session (root) +# POST /v1.0/me/drive/items/{parentId}:/{name}:/createUploadSession +# PUT /v1.0/_upload/{session_id} → strict chunk protocol +# GET /v1.0/me/drive/items/{id}/content → stored bytes verbatim +# +# Colon addressing: the router captures "photo.jpg:" as a segment value; the +# handlers strip (and REQUIRE) the trailing colon, rejecting malformed +# addressing with 400 the way real Graph rejects bad path syntax. +# +# Resumable protocol enforced per the real API: +# - Content-Range: "bytes {start}-{end}/{total}" +# - start must equal the session's next expected offset (sequential, +# contiguous chunks only), end >= start, total consistent across chunks, +# declared range length must match the body; violations → 416. +# - mid-session success → 202 {expirationDateTime, nextExpectedRanges} +# - final range (end == total-1) → assemble, create driveItem, 201. +# - the session is deleted on completion; later chunks → 404. + +_EXPIRATION = "2030-01-01T00:00:00Z" + +# --- simple upload --- + +# on_simple_upload_root handles PUT /v1.0/me/drive/root:/{name}:/content. +def on_simple_upload_root(req): + err = _require_bearer(req) + if err != None: + return err + name = _strip_colon(req["params"]["item"]) + if name == None: + return _err("invalidRequest", 400, "Malformed path addressing: expected root:/{name}:/content.") + return _simple_upload(req, "root", name) + +# on_simple_upload_item handles PUT /v1.0/me/drive/items/{parentId}:/{name}:/content. +def on_simple_upload_item(req): + err = _require_bearer(req) + if err != None: + return err + parent_id = _strip_colon(req["params"]["parent"]) + name = _strip_colon(req["params"]["item"]) + if parent_id == None or name == None: + return _err("invalidRequest", 400, "Malformed path addressing: expected items/{parentId}:/{name}:/content.") + fc = store_collection("files") + parent = fc.get(parent_id) + if parent == None or parent.get("folder") == None: + return _err("itemNotFound", 404, "The parent folder could not be found.") + return _simple_upload(req, parent_id, name) + +# _simple_upload stores the body and creates/replaces the driveItem. +# Real Graph semantics: default PUT to an existing path replaces the content +# (200); conflictBehavior=rename creates a suffixed sibling (201); +# conflictBehavior=fail returns 409. +def _simple_upload(req, parent_id, name): + content = req["raw_body"] + if content == None: + content = "" + content_type = req["headers"].get("Content-Type", "") + if content_type == "": + content_type = "application/octet-stream" + + conflict = req["query"].get("@microsoft.graph.conflictBehavior", "replace") + fc = store_collection("files") + b = store_blob("drive") + + existing = _find_child_by_name(fc, parent_id, name) + if existing != None: + if conflict == "fail": + return _err("nameAlreadyExists", 409, "An item with the same name already exists under the parent.") + if conflict == "rename": + name = _conflict_rename(fc, parent_id, name) + else: + # Replace: same item id, new content. + b.put(existing["id"], content, content_type) + existing["size"] = len(content) + existing["file"] = {"mimeType": content_type} + existing["lastModifiedDateTime"] = "2024-06-15T12:00:00Z" + fc.update(existing["id"], existing) + return respond(200, _drive_item(existing)) + + item_id = _next_item_id() + b.put(item_id, content, content_type) + doc = { + "id": item_id, + "name": name, + "file": {"mimeType": content_type}, + "folder": None, + "size": len(content), + "parentId": parent_id, + "createdDateTime": "2024-06-15T12:00:00Z", + "lastModifiedDateTime": "2024-06-15T12:00:00Z", + } + fc.insert(doc) + return respond(201, _drive_item(doc)) + +# --- path resolution --- + +# on_resolve_path handles GET /v1.0/me/drive/root:/{path}:/ (single-depth), +# supporting ?select=id (and $select=id) to return just the item id. +def on_resolve_path(req): + err = _require_bearer(req) + if err != None: + return err + name = _strip_colon(req["params"]["item"]) + if name == None: + return _err("invalidRequest", 400, "Malformed path addressing: expected root:/{path}:/.") + + fc = store_collection("files") + doc = _find_child_by_name(fc, "root", name) + if doc == None: + return _err("itemNotFound", 404, "The resource could not be found.") + + select = req["query"].get("select", "") + if select == "": + select = req["query"].get("$select", "") + item = _drive_item(doc) + if select != "": + item = _select_fields(item, _split_commas(select)) + return respond(200, item) + +# --- resumable upload sessions --- + +# on_create_session_root handles POST root:/{name}:/createUploadSession. +def on_create_session_root(req): + err = _require_bearer(req) + if err != None: + return err + name = _strip_colon(req["params"]["item"]) + if name == None: + return _err("invalidRequest", 400, "Malformed path addressing: expected root:/{name}:/createUploadSession.") + return _create_session(req, "root", name) + +# on_create_session_item handles POST items/{parentId}:/{name}:/createUploadSession. +def on_create_session_item(req): + err = _require_bearer(req) + if err != None: + return err + parent_id = _strip_colon(req["params"]["parent"]) + name = _strip_colon(req["params"]["item"]) + if parent_id == None or name == None: + return _err("invalidRequest", 400, "Malformed path addressing: expected items/{parentId}:/{name}:/createUploadSession.") + fc = store_collection("files") + parent = fc.get(parent_id) + if parent == None or parent.get("folder") == None: + return _err("itemNotFound", 404, "The parent folder could not be found.") + return _create_session(req, parent_id, name) + +# _create_session mints a session and returns its self-referential uploadUrl +# built from req["host"]. +def _create_session(req, parent_id, name): + body = req["body"] + if body == None: + body = {} + item_props = body.get("item", {}) + if item_props == None: + item_props = {} + conflict = item_props.get("@microsoft.graph.conflictBehavior", "rename") + + session_id = "sess-" + _pad6(store_kv_incr("drive", "session_seq")) + sc = store_collection("sessions") + sc.insert({ + "id": session_id, + "name": name, + "parentId": parent_id, + "conflict": conflict, + "next": 0, + "total": -1, + }) + + return respond(200, { + "uploadUrl": "http://" + req["host"] + "/v1.0/_upload/" + session_id, + "expirationDateTime": _EXPIRATION, + }) + +# on_upload_chunk handles PUT /v1.0/_upload/{session_id}. No bearer check: +# real upload URLs are pre-authenticated. +def on_upload_chunk(req): + session_id = req["params"]["session"] + sc = store_collection("sessions") + sess = sc.get(session_id) + if sess == None: + return _err("itemNotFound", 404, "The upload session was not found or is already completed.") + + parsed = _parse_content_range(req["headers"].get("Content-Range", "")) + if parsed == None: + return _err("invalidRequest", 400, "Missing or malformed Content-Range header (expected 'bytes {start}-{end}/{total}').") + start = parsed[0] + end = parsed[1] + total = parsed[2] + + if end < start: + return _range_err("Range end precedes range start.") + if sess["total"] >= 0 and total != sess["total"]: + return _range_err("Total size differs from earlier chunks.") + if end >= total: + return _range_err("Range end exceeds the declared total size.") + if start != sess["next"]: + return _range_err("Chunk start does not match the next expected offset " + str(sess["next"]) + ". Chunks must be sequential and contiguous.") + + content = req["raw_body"] + if content == None: + content = "" + if len(content) != end - start + 1: + return _range_err("Body length does not match the declared Content-Range.") + + b = store_blob("drive") + partial = "" + if start > 0: + existing = b.get("up-" + session_id) + if existing != None: + partial = existing + partial = partial + content + b.put("up-" + session_id, partial, "application/octet-stream") + + if end == total - 1: + # Final range: assemble the driveItem, honoring the session's + # conflict behavior, and invalidate the session. + fc = store_collection("files") + name = sess["name"] + parent_id = sess["parentId"] + conflict = sess.get("conflict", "rename") + existing_item = _find_child_by_name(fc, parent_id, name) + if existing_item != None: + if conflict == "fail": + b.delete("up-" + session_id) + sc.delete(session_id) + return _err("nameAlreadyExists", 409, "An item with the same name already exists under the parent.") + if conflict == "replace": + b.put(existing_item["id"], partial, "application/octet-stream") + existing_item["size"] = len(partial) + existing_item["file"] = {"mimeType": "application/octet-stream"} + fc.update(existing_item["id"], existing_item) + b.delete("up-" + session_id) + sc.delete(session_id) + return respond(200, _drive_item(existing_item)) + name = _conflict_rename(fc, parent_id, name) + + item_id = _next_item_id() + b.put(item_id, partial, "application/octet-stream") + doc = { + "id": item_id, + "name": name, + "file": {"mimeType": "application/octet-stream"}, + "folder": None, + "size": len(partial), + "parentId": parent_id, + "createdDateTime": "2024-06-15T12:00:00Z", + "lastModifiedDateTime": "2024-06-15T12:00:00Z", + } + fc.insert(doc) + b.delete("up-" + session_id) + sc.delete(session_id) + return respond(201, _drive_item(doc)) + + # Mid-session: record progress, report the next expected offset. + sess["next"] = end + 1 + sess["total"] = total + sc.update(session_id, sess) + return respond(202, { + "expirationDateTime": _EXPIRATION, + "nextExpectedRanges": [str(end + 1) + "-"], + }) + +# --- content download --- + +# on_get_content serves the stored bytes verbatim. +# GET /v1.0/me/drive/items/{id}/content (Bearer) +def on_get_content(req): + err = _require_bearer(req) + if err != None: + return err + + item_id = req["params"]["id"] + fc = store_collection("files") + doc = fc.get(item_id) + if doc == None or doc.get("folder") != None: + return _err("itemNotFound", 404, "The resource could not be found.") + + b = store_blob("drive") + content = b.get(item_id) + if content == None: + return _err("itemNotFound", 404, "The item has no stored content.") + content_type = "application/octet-stream" + info = b.stat(item_id) + if info != None: + ct = info.get("content_type", "") + if ct != "" and ct != None: + content_type = ct + return respond(200, content, {"Content-Type": content_type}) + +# --- helpers --- + +# _parse_content_range parses "bytes {start}-{end}/{total}" into a +# (start, end, total) tuple, or None when malformed. +def _parse_content_range(h): + if h == None or not h.startswith("bytes "): + return None + rest = h[6:] + dash = rest.find("-") + slash = rest.find("/") + if dash < 0 or slash < 0 or slash < dash: + return None + start_s = rest[:dash].strip() + end_s = rest[dash + 1:slash].strip() + total_s = rest[slash + 1:].strip() + if not _is_digits(start_s) or not _is_digits(end_s) or not _is_digits(total_s): + return None + return (_to_int(start_s), _to_int(end_s), _to_int(total_s)) + +# _range_err returns the 416 invalidRange error used for every resumable +# protocol violation. +def _range_err(detail): + return _err("invalidRange", 416, "The Content-Range is not valid for this session: " + detail) diff --git a/adapters/microsoft-graph-style/scripts/lib.star b/adapters/microsoft-graph-style/scripts/lib.star index c839d70..196f192 100644 --- a/adapters/microsoft-graph-style/scripts/lib.star +++ b/adapters/microsoft-graph-style/scripts/lib.star @@ -205,6 +205,89 @@ def _apply_odata(entities, query, base_url): envelope["@odata.nextLink"] = _odata_link(base_url, skip + top) return respond(200, envelope) +# --- OneDrive driveItem helpers (shared by drive.star and drive_upload.star) --- + +_DRIVE_ID = "b!mock-drive-id-0001" + +# _is_digits reports whether s is a non-empty run of ASCII digits. +def _is_digits(s): + if s == None or len(s) == 0: + return False + for i in range(len(s)): + ch = s[i] + if ch < "0" or ch > "9": + return False + return True + +# _next_item_id mints a monotonic driveItem id. +def _next_item_id(): + return "item-" + _pad6(store_kv_incr("drive", "item_seq")) + +# _strip_colon strips the trailing ':' of a colon-addressed path segment +# ("photo.jpg:" → "photo.jpg"). Returns None if the segment does not end +# with ':' — the caller should reject the request as malformed addressing. +def _strip_colon(seg): + if seg == None or len(seg) < 2 or seg[-1:] != ":": + return None + return seg[:-1] + +# _find_child_by_name returns the doc under parent_id with the given name, +# or None. +def _find_child_by_name(fc, parent_id, name): + for d in fc.list(): + if d.get("parentId", "root") == parent_id and d.get("name", "") == name: + return d + return None + +# _conflict_rename returns the first free " (n)"-suffixed variant of name +# under parent_id ("photo.jpg" → "photo (1).jpg"). +def _conflict_rename(fc, parent_id, name): + dot = name.rfind(".") + stem = name + ext = "" + if dot > 0: + stem = name[:dot] + ext = name[dot:] + n = 1 + for _ in range(1000): + candidate = stem + " (" + str(n) + ")" + ext + if _find_child_by_name(fc, parent_id, candidate) == None: + return candidate + n = n + 1 + return stem + " (" + str(n) + ")" + ext + +# _parent_ref builds the parentReference facet for a parentId. +def _parent_ref(parent_id): + path = "/drive/root:" + if parent_id != "root": + fc = store_collection("files") + parent = fc.get(parent_id) + if parent != None: + path = "/drive/root:/" + parent.get("name", "") + return { + "driveId": _DRIVE_ID, + "driveType": "business", + "id": parent_id, + "path": path, + } + +# _drive_item builds the public driveItem JSON from a stored files doc. +# Absent facets (file/folder) are omitted, matching real Graph responses. +def _drive_item(doc): + item = { + "id": doc["id"], + "name": doc["name"], + "size": doc.get("size", 0), + "parentReference": _parent_ref(doc.get("parentId", "root")), + "createdDateTime": doc.get("createdDateTime", "2024-01-01T00:00:00Z"), + "lastModifiedDateTime": doc.get("lastModifiedDateTime", "2024-01-01T00:00:00Z"), + } + if doc.get("file") != None: + item["file"] = doc["file"] + if doc.get("folder") != None: + item["folder"] = doc["folder"] + return item + # _me returns the constant mock "me" profile used by /me and as the sender # for mail/calendar. This mock uses a fixed identity so tests can assert # stable fields. diff --git a/adapters/photos-style/README.md b/adapters/photos-style/README.md index 2030e4d..7990f55 100644 --- a/adapters/photos-style/README.md +++ b/adapters/photos-style/README.md @@ -16,16 +16,31 @@ two-step upload pipeline: - **OAuth2:** authorize redirect + token exchange + refresh (shared Google OAuth2 flow, self-contained). -- **Uploads:** `POST /v1/uploads` (raw octet-stream) → uploadToken (plain text). +- **Uploads:** `POST /v1/uploads` (raw octet-stream) → uploadToken (plain + text). The raw bytes are stored in the blob store keyed by the token, and + the request Content-Type is recorded. - **batchCreate:** `POST /v1/mediaItems:batchCreate` ({albumId?, newMediaItems:[{description, simpleMediaItem:{uploadToken, fileName}}]}) → `{newMediaItemResults:[{mediaItem:{id, productUrl, baseUrl, mimeType, - filename, mediaMetadata}}]}`. -- **Search:** `POST /v1/mediaItems:search` → `{mediaItems:[...]}` — STATEFUL: - items created via batchCreate appear in search. -- **List:** `GET /v1/mediaItems` → `{mediaItems:[...]}`. + filename, mediaMetadata}}]}`. Links the uploaded bytes to the created + media item. +- **Search:** `POST /v1/mediaItems:search` → `{mediaItems:[...], + nextPageToken?}` — STATEFUL: items created via batchCreate appear in + search. Honors `pageSize`/`pageToken` from the JSON body. +- **List:** `GET /v1/mediaItems` → `{mediaItems:[...], nextPageToken?}`. + Honors `pageSize`/`pageToken` query parameters. +- **Get:** `GET /v1/mediaItems/{id}` → the public media item. +- **Media download:** `GET /v1/media-dl/{id}` serves the media bytes (the + `baseUrl` target). Strict Google suffix semantics: `{id}=d` / `{id}=dv` + serve the ORIGINAL uploaded bytes; a bare `{id}` serves a deterministic + derivative payload with different bytes, so clients that forget the + suffix fail byte-comparison loudly. - **Albums:** list, create, and get album details. +`baseUrl` is computed at read time from the request Host header +(`http://{host}/v1/media-dl/{id}`), so responses always point back at the +address the client used to reach the simulator. + State persists in SQLite-backed collections, so media items and albums created in one request are visible in subsequent requests within the same `stunt up` session. @@ -38,8 +53,10 @@ session. | POST | `/o/oauth2/token` | `oauth.star#on_token` | Token exchange (auth code + refresh) | | POST | `/v1/uploads` | `uploads.star#on_uploads` | Raw octet-stream → uploadToken | | POST | `/v1/mediaItems:batchCreate` | `media_items.star#on_batch_create` | Create media items from tokens | -| POST | `/v1/mediaItems:search` | `media_items.star#on_search` | Search media items (stateful) | -| GET | `/v1/mediaItems` | `media_items.star#on_list` | List media items | +| POST | `/v1/mediaItems:search` | `media_items.star#on_search` | Search media items (stateful, paginated) | +| GET | `/v1/mediaItems` | `media_items.star#on_list` | List media items (paginated) | +| GET | `/v1/mediaItems/{id}` | `media_items.star#on_get` | Get one media item | +| GET | `/v1/media-dl/{id}` | `media_items.star#on_media_dl` | Media bytes (`=d`/`=dv` original, bare derivative) | | GET | `/v1/albums` | `albums.star#on_list_albums` | List albums | | POST | `/v1/albums` | `albums.star#on_create_album` | Create album | | GET | `/v1/albums/{id}` | `albums.star#on_get_album` | Get album details | @@ -79,6 +96,11 @@ Point a `stunt.yaml` service at this directory: services: photos: adapter: ./adapters/photos-style + max_body_bytes: 33554432 # uploads over 1 MiB need a raised body limit ``` Then `stunt up` and make requests to the served address. + +Note: the engine's default request-body limit is 1 MiB and oversize bodies +get a `413`. Real phone photos routinely run 3-8 MB, so set +`max_body_bytes` on the service (as above) when testing realistic uploads. diff --git a/adapters/photos-style/adapter.yaml b/adapters/photos-style/adapter.yaml index 9135475..da45bea 100644 --- a/adapters/photos-style/adapter.yaml +++ b/adapters/photos-style/adapter.yaml @@ -39,6 +39,18 @@ endpoints: - route: /v1/mediaItems method: GET handler: scripts/media_items.star#on_list + - route: /v1/mediaItems/{id} + method: GET + handler: scripts/media_items.star#on_get + + # --- Media download (baseUrl target) --- + # The {id} segment may carry the Google download suffix as part of the + # captured value (e.g. "mock-media-1=d" or "mock-media-1=dv"): the router + # matches '=' inside a param. Strict semantics: only =d/=dv serve the + # original bytes; a bare baseUrl serves a distinct derivative payload. + - route: /v1/media-dl/{id} + method: GET + handler: scripts/media_items.star#on_media_dl # --- Albums --- - route: /v1/albums diff --git a/adapters/photos-style/scripts/lib.star b/adapters/photos-style/scripts/lib.star index 02b1c01..07768ec 100644 --- a/adapters/photos-style/scripts/lib.star +++ b/adapters/photos-style/scripts/lib.star @@ -58,3 +58,19 @@ def _to_num(v, default=0): # _contains reports whether substr appears within s. def _contains(s, substr): return s.find(substr) >= 0 + +# _paginate slices items by pageSize/pageToken (an offset-based cursor) and +# returns (page, next_token). next_token is None when no items remain. +def _paginate(items, page_size, page_token): + start = _to_num(page_token, 0) + if start < 0 or start > len(items): + start = len(items) + if page_size <= 0: + page_size = len(items) + end = start + page_size + if end > len(items): + end = len(items) + next_token = None + if end < len(items): + next_token = str(end) + return items[start:end], next_token diff --git a/adapters/photos-style/scripts/media_items.star b/adapters/photos-style/scripts/media_items.star index cfa0aff..bd06930 100644 --- a/adapters/photos-style/scripts/media_items.star +++ b/adapters/photos-style/scripts/media_items.star @@ -1,14 +1,29 @@ -# Media items handlers — batchCreate, search, list. +# Media items handlers — batchCreate, search, list, get, media download. # # POST /v1/mediaItems:batchCreate (Bearer; JSON) -> { newMediaItemResults: [...] } -# STATEFUL: created items appear in search. +# STATEFUL: created items appear in search; uploaded bytes are linked to +# the created item in the blob store. # POST /v1/mediaItems:search (Bearer; JSON) -> { mediaItems: [...], nextPageToken? } +# Honors pageSize/pageToken from the JSON body. # GET /v1/mediaItems (Bearer) -> { mediaItems: [...], nextPageToken? } +# Honors pageSize/pageToken query parameters. +# GET /v1/mediaItems/{id} (Bearer) -> the public media item +# GET /v1/media-dl/{id} -> media bytes (baseUrl target; no auth, like the +# real pre-authorized baseUrl). STRICT suffix semantics: "{id}=d" / "{id}=dv" +# serve the ORIGINAL uploaded bytes; a bare "{id}" serves a deterministic +# DERIVATIVE payload with clearly different bytes so clients that forget +# the suffix fail byte-comparison loudly. +# +# baseUrl is computed AT READ TIME from req["host"]; no host is stored in +# the documents, so responses always point at the address the client used. + +# Shared helpers (_bearer, _user_for_token, _to_num, _paginate) are preloaded +# from scripts/lib.star. -# Shared helpers (_bearer, _user_for_token, _to_int) are preloaded from -# scripts/lib.star. +_DEFAULT_PAGE_SIZE = 25 -# on_batch_create creates media items from upload tokens. +# on_batch_create creates media items from upload tokens and links the +# uploaded bytes (stored under the uploadToken) to the new media item id. def on_batch_create(req): user = _user_for_token(req) if user == None: @@ -25,6 +40,7 @@ def on_batch_create(req): utc = store_collection("upload_tokens") mc = store_collection("media_items") + b = store_blob("photos") results = [] for item in new_media_items: @@ -42,10 +58,18 @@ def on_batch_create(req): media_seq = store_kv_incr("photos", "media_seq") media_id = "mock-media-" + str(media_seq) + # Link the uploaded bytes to the media item: copy the blob from + # the uploadToken key to the media id, keeping the recorded + # Content-Type. + content = b.get(upload_token) + if content == None: + content = "" + content_type = tok_doc.get("content_type", "application/octet-stream") + b.put(media_id, content, content_type) + media_item = { "id": media_id, "productUrl": "https://photos.google.com/mock/" + media_id, - "baseUrl": "https://mock-photos.example/base/" + media_id, "mimeType": _guess_mime(file_name), "filename": file_name, "mediaMetadata": { @@ -59,15 +83,16 @@ def on_batch_create(req): } mc.insert(media_item) - status = {"mediaItem": media_item} + status = {"mediaItem": _public_media_item(media_item, req["host"])} else: status = {"status": {"code": 3, "message": "Invalid upload token"}} results.append(status) return respond(200, {"newMediaItemResults": results}) -# on_search searches media items. Returns all items (optionally filtered by -# albumId). STATEFUL: items from batchCreate appear here. +# on_search searches media items with pagination. Returns the caller's items +# (optionally filtered by albumId). STATEFUL: items from batchCreate appear +# here. pageSize/pageToken come from the JSON body. def on_search(req): user = _user_for_token(req) if user == None: @@ -78,7 +103,8 @@ def on_search(req): body = {} album_id = body.get("albumId", "") - page_size = _to_num(body.get("pageSize", 25), 25) + page_size = _to_num(body.get("pageSize", _DEFAULT_PAGE_SIZE), _DEFAULT_PAGE_SIZE) + page_token = body.get("pageToken", "") mc = store_collection("media_items") all_items = mc.list() @@ -88,34 +114,96 @@ def on_search(req): continue if album_id != "" and doc.get("albumId", "") != album_id: continue - items.append(_public_media_item(doc)) + items.append(_public_media_item(doc, req["host"])) - result = {"mediaItems": items} + page, next_token = _paginate(items, page_size, page_token) + result = {"mediaItems": page} + if next_token != None: + result["nextPageToken"] = next_token return respond(200, result) -# on_list lists all media items for the authenticated user. +# on_list lists media items for the authenticated user with pagination. +# pageSize/pageToken come from the query string. def on_list(req): user = _user_for_token(req) if user == None: return respond(401, {"error": {"code": 401, "message": "Invalid credentials", "status": "UNAUTHENTICATED"}}) + page_size = _to_num(req["query"].get("pageSize", ""), _DEFAULT_PAGE_SIZE) + page_token = req["query"].get("pageToken", "") + mc = store_collection("media_items") all_items = mc.list() items = [] for doc in all_items: if doc.get("user", "") != user["sub"]: continue - items.append(_public_media_item(doc)) + items.append(_public_media_item(doc, req["host"])) - result = {"mediaItems": items} + page, next_token = _paginate(items, page_size, page_token) + result = {"mediaItems": page} + if next_token != None: + result["nextPageToken"] = next_token return respond(200, result) -# _public_media_item strips internal fields (user, albumId) from a stored doc. -def _public_media_item(doc): +# on_get returns a single public media item by id. +# GET /v1/mediaItems/{id} (Bearer) +def on_get(req): + user = _user_for_token(req) + if user == None: + return respond(401, {"error": {"code": 401, "message": "Invalid credentials", "status": "UNAUTHENTICATED"}}) + + media_id = req["params"]["id"] + mc = store_collection("media_items") + doc = mc.get(media_id) + if doc == None or doc.get("user", "") != user["sub"]: + return respond(404, {"error": {"code": 404, "message": "Media item not found", "status": "NOT_FOUND"}}) + return respond(200, _public_media_item(doc, req["host"])) + +# on_media_dl serves media bytes for a baseUrl. No auth: the real baseUrl is +# a pre-authorized URL. The {id} param may carry the download suffix inside +# the captured segment ("mock-media-1=d" / "mock-media-1=dv"). +def on_media_dl(req): + raw = req["params"]["id"] + original = False + media_id = raw + if raw.endswith("=dv"): + media_id = raw[:-3] + original = True + elif raw.endswith("=d"): + media_id = raw[:-2] + original = True + + mc = store_collection("media_items") + doc = mc.get(media_id) + if doc == None: + return respond(404, {"error": {"code": 404, "message": "Media item not found", "status": "NOT_FOUND"}}) + + b = store_blob("photos") + if original: + content = b.get(media_id) + if content == None: + return respond(404, {"error": {"code": 404, "message": "Media bytes not found", "status": "NOT_FOUND"}}) + info = b.stat(media_id) + content_type = "" + if info != None: + content_type = info.get("content_type", "") + if content_type == "" or content_type == None: + content_type = "application/octet-stream" + return respond(200, content, {"Content-Type": content_type}) + + # Bare baseUrl: deterministic derivative payload, clearly different bytes + # from the original, so a missing =d suffix fails byte-equality loudly. + payload = "stunt-derivative-preview:" + media_id + ":not-the-original-bytes" + return respond(200, payload, {"Content-Type": "image/jpeg"}) + +# _public_media_item builds the public shape from a stored doc, computing +# baseUrl at read time from the request host. +def _public_media_item(doc, host): return { "id": doc["id"], "productUrl": doc["productUrl"], - "baseUrl": doc["baseUrl"], + "baseUrl": "http://" + host + "/v1/media-dl/" + doc["id"], "mimeType": doc["mimeType"], "filename": doc["filename"], "mediaMetadata": doc["mediaMetadata"], diff --git a/adapters/photos-style/scripts/uploads.star b/adapters/photos-style/scripts/uploads.star index 69959d1..5ed0f7b 100644 --- a/adapters/photos-style/scripts/uploads.star +++ b/adapters/photos-style/scripts/uploads.star @@ -6,25 +6,38 @@ # 1. POST /v1/uploads with raw binary → returns an uploadToken (plain text) # 2. POST /v1/mediaItems:batchCreate with the uploadToken → creates a mediaItem # -# This handler mints and stores an uploadToken so that batchCreate can -# reference it. The raw body is not JSON (octet-stream), so req["body"] is -# None — that's expected; we just mint the token. +# This handler mints an uploadToken AND stores the raw request bytes in the +# blob store keyed by that token, so batchCreate can link the bytes to the +# created media item and /v1/media-dl/{id} can serve them back byte-exact. +# The request Content-Type is recorded alongside the token. # Shared helpers (_bearer, _user_for_token) are preloaded from scripts/lib.star. -# on_uploads mints an uploadToken for the uploaded bytes. +# on_uploads mints an uploadToken and stores the uploaded bytes. def on_uploads(req): user = _user_for_token(req) if user == None: return respond(401, {"error": {"code": 401, "message": "Invalid credentials", "status": "UNAUTHENTICATED"}}) seq = store_kv_incr("photos", "upload_seq") + # Generated id only (blob names must be [A-Za-z0-9][A-Za-z0-9._-]*). token = "CAISI" + str(seq) + "mockUploadToken" + content = req["raw_body"] + if content == None: + content = "" + content_type = req["headers"].get("Content-Type", "") + if content_type == "": + content_type = "application/octet-stream" + + b = store_blob("photos") + b.put(token, content, content_type) + utc = store_collection("upload_tokens") utc.insert({ "id": token, "user": user["sub"], + "content_type": content_type, }) # uploadToken is returned as plain text (not JSON). diff --git a/internal/cli/llm.go b/internal/cli/llm.go index 5712e45..5c0150b 100644 --- a/internal/cli/llm.go +++ b/internal/cli/llm.go @@ -67,6 +67,7 @@ Global: --manifest (default stunt.yaml). Cache: --cache-dir/$STUNT_ADAPTE adapter: ./adapters/svc-style # local path OR git:github.com/org/repo@ref # OR embedded:stripe-style (bundled in binary, no clone) config: { webhook_url: ... } # optional, passed to scripts + max_body_bytes: 8388608 # optional request-body cap (default 1 MiB); oversize -> 413 # OR rules-only (no adapter): # rules: # - name: ok @@ -88,7 +89,8 @@ Rules match in ORDER (first wins); "/**" is the catch-all. Templates: {{ faker.E # optional transports: grpc: {service, descriptor, methods[]} | graphql: {schema, resolvers, path} | ws: [{route, handler}] ## Starlark handler API (def on_x(req): ... return respond(...)) -req keys: method, path, headers (dict, case-insensitive), body (parsed JSON or None), +req keys: method, path, host (request Host header, for self-referential URLs), + headers (dict, case-insensitive), body (parsed JSON or None), raw_body (string, for binary), query (dict), params (path captures). Builtins: respond(status, body, headers) # or return {status, body, headers} diff --git a/internal/engine/adapter_dispatch.go b/internal/engine/adapter_dispatch.go index 6cb3420..791d7c8 100644 --- a/internal/engine/adapter_dispatch.go +++ b/internal/engine/adapter_dispatch.go @@ -135,6 +135,7 @@ func (e *Engine) runHandler( req := starlark.Request{ Method: r.Method, Path: r.URL.Path, + Host: r.Host, Headers: headerMap(r.Header), Body: bodyMap, RawBody: string(body), diff --git a/internal/engine/body_limit_test.go b/internal/engine/body_limit_test.go new file mode 100644 index 0000000..14a4466 --- /dev/null +++ b/internal/engine/body_limit_test.go @@ -0,0 +1,162 @@ +package engine + +import ( + "bytes" + "context" + "io" + "net/http" + "testing" + "time" + + "stuntapi.com/stunt/internal/manifest" +) + +// echoAdapterYAML is a minimal adapter whose single handler echoes the raw +// request body back verbatim, so tests can assert byte-exact round-trips. +const echoAdapterYAML = ` +id: echo +name: Echo +endpoints: + - route: /echo + method: POST + handler: scripts/echo.star#on_echo +` + +const echoStar = ` +def on_echo(req): + return respond(200, req["raw_body"], {"Content-Type": "application/octet-stream"}) +` + +// bootEchoService builds an in-test echo adapter, boots an engine with the +// given per-service max_body_bytes (0 = default), and returns the base URL. +func bootEchoService(t *testing.T, maxBodyBytes int64) string { + t.Helper() + + adapterDir := t.TempDir() + writeFile(t, adapterDir, "adapter.yaml", echoAdapterYAML) + writeFile(t, adapterDir, "scripts/echo.star", echoStar) + + stateDir := t.TempDir() + m := &manifest.Manifest{ + Path: stateDir + "/stunt.yaml", + Version: 1, + Network: manifest.Network{Mode: "port", BasePort: 0}, + Services: map[string]manifest.Service{ + "echo": {Adapter: adapterDir, MaxBodyBytes: maxBodyBytes}, + }, + } + + e, err := New(m) + if err != nil { + t.Fatalf("engine.New: %v", err) + } + t.Cleanup(func() { e.Close() }) + + addrs, cancel, err := e.ServeForTest(context.Background()) + if err != nil { + t.Fatalf("ServeForTest: %v", err) + } + t.Cleanup(cancel) + time.Sleep(50 * time.Millisecond) + + return addrs["echo"] +} + +// postBytes POSTs raw bytes and returns the response status and body. +func postBytes(t *testing.T, url string, payload []byte) (int, []byte) { + t.Helper() + req, err := http.NewRequest("POST", url, bytes.NewReader(payload)) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Content-Type", "application/octet-stream") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + body, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + return resp.StatusCode, body +} + +// allByteValues returns a payload that contains every possible byte value +// (0x00-0xFF), prefixed with the PNG magic so it is definitely not valid +// UTF-8 or JSON. Any sanitization anywhere in the pipeline breaks the +// byte-equality assertions loudly. +func allByteValues(prefixLen int) []byte { + png := []byte{0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'} + out := append([]byte{}, png...) + for i := 0; i < 256; i++ { + out = append(out, byte(i)) + } + for len(out) < prefixLen { + out = append(out, byte(len(out)%251)) + } + return out +} + +// TestBodyLimitDefaultOversizeIs413 proves the engine no longer silently +// truncates bodies over the default 1 MiB limit: the request must fail with +// 413, and the handler must never see truncated data. +func TestBodyLimitDefaultOversizeIs413(t *testing.T) { + base := bootEchoService(t, 0) + + payload := bytes.Repeat([]byte{0xAB}, 1<<20+1) // 1 MiB + 1 + status, _ := postBytes(t, base+"/echo", payload) + if status != http.StatusRequestEntityTooLarge { + t.Fatalf("oversize body -> status %d, want 413", status) + } +} + +// TestBodyLimitDefaultUnderLimitRoundTrips proves a body just under the +// default limit round-trips byte-exact (no truncation, no mangling). +func TestBodyLimitDefaultUnderLimitRoundTrips(t *testing.T) { + base := bootEchoService(t, 0) + + payload := allByteValues(1<<20 - 1) // 1 byte under the default limit + status, body := postBytes(t, base+"/echo", payload) + if status != 200 { + t.Fatalf("under-limit body -> status %d, want 200", status) + } + if !bytes.Equal(body, payload) { + t.Fatalf("echoed body differs from sent body: got %d bytes, want %d bytes (byte-exact)", len(body), len(payload)) + } +} + +// TestBodyLimitConfigurable proves max_body_bytes raises and lowers the +// per-service limit. +func TestBodyLimitConfigurable(t *testing.T) { + t.Run("small limit rejects oversize", func(t *testing.T) { + base := bootEchoService(t, 128) + + status, _ := postBytes(t, base+"/echo", bytes.Repeat([]byte{0x01}, 129)) + if status != http.StatusRequestEntityTooLarge { + t.Fatalf("129 bytes with 128-byte limit -> status %d, want 413", status) + } + + payload := bytes.Repeat([]byte{0x02}, 128) + status, body := postBytes(t, base+"/echo", payload) + if status != 200 { + t.Fatalf("128 bytes with 128-byte limit -> status %d, want 200", status) + } + if !bytes.Equal(body, payload) { + t.Fatalf("at-limit body did not round-trip byte-exact") + } + }) + + t.Run("raised limit accepts multi-MiB body", func(t *testing.T) { + base := bootEchoService(t, 8<<20) + + payload := allByteValues(3 << 20) // 3 MiB, over the old 1 MiB default + status, body := postBytes(t, base+"/echo", payload) + if status != 200 { + t.Fatalf("3 MiB body with 8 MiB limit -> status %d, want 200", status) + } + if !bytes.Equal(body, payload) { + t.Fatalf("3 MiB body did not round-trip byte-exact: got %d bytes, want %d", len(body), len(payload)) + } + }) +} diff --git a/internal/engine/engine.go b/internal/engine/engine.go index 57821f3..9b866f8 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -3,6 +3,7 @@ package engine import ( "context" "crypto/sha256" + "errors" "fmt" "io" "log" @@ -402,12 +403,20 @@ func (e *Engine) HTTPServerForTest() *http.Server { return &http.Server{Handler: e.HandlerForTest(), ReadHeaderTimeout: 5 * time.Second} } +// defaultMaxBodyBytes is the request body cap applied when a service does +// not set max_body_bytes in the manifest. +const defaultMaxBodyBytes = 1 << 20 // 1 MiB + func (e *Engine) serviceHandler(name string, svc manifest.Service) http.Handler { rng := rules.NewRNG(e.manifest.RNGSeed) fk := rules.NewFaker(e.manifest.RNGSeed) baseDir := filepath.Dir(e.manifest.Path) st := e.states[name] // nil for rules-only services loadErr := e.loadErrors[name] // non-empty if adapter failed to load + bodyLimit := svc.MaxBodyBytes + if bodyLimit <= 0 { + bodyLimit = defaultMaxBodyBytes + } // rng and faker are shared across goroutines; math/rand.Rand and gofakeit // are not concurrency-safe. Guard all access with a mutex (I2). @@ -444,7 +453,20 @@ func (e *Engine) serviceHandler(name string, svc manifest.Service) http.Handler var body []byte if r.Body != nil { - body, _ = io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20)) + var err error + body, err = io.ReadAll(http.MaxBytesReader(w, r.Body, bodyLimit)) + if err != nil { + // Never hand truncated data to a handler. Overflow is 413; + // any other read failure is a 400. + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + writeStatus(w, http.StatusRequestEntityTooLarge, + fmt.Sprintf(`{"error":"request body exceeds %d bytes"}`, bodyLimit)) + return + } + writeStatus(w, http.StatusBadRequest, `{"error":"failed to read request body"}`) + return + } } // --- adapter-backed dispatch --- diff --git a/internal/engine/graph_drive_write_test.go b/internal/engine/graph_drive_write_test.go new file mode 100644 index 0000000..0745ff4 --- /dev/null +++ b/internal/engine/graph_drive_write_test.go @@ -0,0 +1,528 @@ +package engine + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "path/filepath" + "strings" + "testing" + "time" + + "stuntapi.com/stunt/internal/manifest" +) + +// bootGraphService boots the microsoft-graph-style reference adapter on a +// free port with an optional max_body_bytes and returns its base URL. +func bootGraphService(t *testing.T, maxBodyBytes int64) string { + t.Helper() + + adapterDir, err := filepath.Abs(filepath.Join("..", "..", "adapters", "microsoft-graph-style")) + if err != nil { + t.Fatal(err) + } + stateDir := t.TempDir() + m := &manifest.Manifest{ + Path: filepath.Join(stateDir, "stunt.yaml"), + Version: 1, + Network: manifest.Network{Mode: "port", BasePort: 0}, + Services: map[string]manifest.Service{ + "graph": {Adapter: adapterDir, MaxBodyBytes: maxBodyBytes}, + }, + } + e, err := New(m) + if err != nil { + t.Fatalf("engine.New: %v", err) + } + t.Cleanup(func() { e.Close() }) + addrs, cancel, err := e.ServeForTest(context.Background()) + if err != nil { + t.Fatalf("ServeForTest: %v", err) + } + t.Cleanup(cancel) + time.Sleep(50 * time.Millisecond) + return addrs["graph"] +} + +// graphDo performs an HTTP request with optional bearer auth and body, and +// returns status, body bytes, and the response. +func graphDo(t *testing.T, method, url, token string, payload []byte, headers map[string]string) (int, []byte) { + t.Helper() + var rd io.Reader + if payload != nil { + rd = bytes.NewReader(payload) + } + req, err := http.NewRequest(method, url, rd) + if err != nil { + t.Fatal(err) + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + for k, v := range headers { + req.Header.Set(k, v) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + b, _ := io.ReadAll(resp.Body) + return resp.StatusCode, b +} + +// graphJSON unmarshals a JSON object body or fails the test. +func graphJSON(t *testing.T, b []byte) map[string]any { + t.Helper() + var out map[string]any + if err := json.Unmarshal(b, &out); err != nil { + t.Fatalf("unmarshal %q: %v", b, err) + } + return out +} + +const graphToken = "mock-bearer-token" + +// TestGraphDriveSimpleUpload proves the simple upload path (files under +// 4 MB): PUT root:/{name}:/content stores the bytes, creates a driveItem, +// and the content round-trips byte-exact through GET items/{id}/content. +func TestGraphDriveSimpleUpload(t *testing.T) { + base := bootGraphService(t, 0) + + original := allByteValues(2048) + + // ===== PUT /v1.0/me/drive/root:/fixture.bin:/content -> 201 driveItem ===== + + status, body := graphDo(t, "PUT", base+"/v1.0/me/drive/root:/fixture.bin:/content", graphToken, + original, map[string]string{"Content-Type": "application/octet-stream"}) + if status != 201 { + t.Fatalf("simple upload -> status %d, want 201; body %s", status, body) + } + item := graphJSON(t, body) + itemID, _ := item["id"].(string) + if itemID == "" { + t.Fatalf("driveItem.id missing: %s", body) + } + if item["name"] != "fixture.bin" { + t.Fatalf("driveItem.name = %v, want fixture.bin", item["name"]) + } + if int64(item["size"].(float64)) != int64(len(original)) { + t.Fatalf("driveItem.size = %v, want %d", item["size"], len(original)) + } + parentRef, ok := item["parentReference"].(map[string]any) + if !ok { + t.Fatalf("driveItem.parentReference = %v, want object", item["parentReference"]) + } + if parentRef["id"] != "root" { + t.Fatalf("parentReference.id = %v, want root", parentRef["id"]) + } + + // ===== GET /v1.0/me/drive/items/{id}/content -> byte-exact ===== + + status, got := graphDo(t, "GET", base+"/v1.0/me/drive/items/"+itemID+"/content", graphToken, nil, nil) + if status != 200 { + t.Fatalf("get content -> status %d, want 200", status) + } + if !bytes.Equal(got, original) { + t.Fatalf("content is not byte-equal: got %d bytes, want %d", len(got), len(original)) + } + + // ===== default PUT to the same name replaces (200, same id) ===== + + replaced := []byte("replaced-content-bytes") + status, body = graphDo(t, "PUT", base+"/v1.0/me/drive/root:/fixture.bin:/content", graphToken, + replaced, map[string]string{"Content-Type": "application/octet-stream"}) + if status != 200 { + t.Fatalf("replace upload -> status %d, want 200; body %s", status, body) + } + rep := graphJSON(t, body) + if rep["id"] != itemID { + t.Fatalf("replace changed the item id: %v, want %v", rep["id"], itemID) + } + status, got = graphDo(t, "GET", base+"/v1.0/me/drive/items/"+itemID+"/content", graphToken, nil, nil) + if status != 200 || !bytes.Equal(got, replaced) { + t.Fatalf("replaced content mismatch: status %d, got %q", status, got) + } + + // ===== conflictBehavior=rename creates "fixture (1).bin" ===== + + renamed := []byte("renamed-copy-bytes") + status, body = graphDo(t, "PUT", + base+"/v1.0/me/drive/root:/fixture.bin:/content?@microsoft.graph.conflictBehavior=rename", + graphToken, renamed, map[string]string{"Content-Type": "application/octet-stream"}) + if status != 201 { + t.Fatalf("rename upload -> status %d, want 201; body %s", status, body) + } + ren := graphJSON(t, body) + if ren["name"] != "fixture (1).bin" { + t.Fatalf("renamed item name = %v, want %q", ren["name"], "fixture (1).bin") + } + if ren["id"] == itemID { + t.Fatal("rename reused the existing item id; want a new item") + } + + // ===== conflictBehavior=fail -> 409 ===== + + status, body = graphDo(t, "PUT", + base+"/v1.0/me/drive/root:/fixture.bin:/content?@microsoft.graph.conflictBehavior=fail", + graphToken, []byte("x"), nil) + if status != 409 { + t.Fatalf("fail-on-conflict upload -> status %d, want 409; body %s", status, body) + } + + // ===== malformed addressing (no trailing colon) -> 400 ===== + + status, _ = graphDo(t, "PUT", base+"/v1.0/me/drive/root:/fixture.bin/content", graphToken, []byte("x"), nil) + if status != 400 { + t.Fatalf("missing trailing colon -> status %d, want 400", status) + } + + // ===== no auth -> 401 ===== + + status, _ = graphDo(t, "PUT", base+"/v1.0/me/drive/root:/other.bin:/content", "", []byte("x"), nil) + if status != 401 { + t.Fatalf("simple upload without auth -> status %d, want 401", status) + } +} + +// TestGraphDriveFoldersAndResolution proves createFolder, per-parent child +// listing, path resolution with select=id, and the folder upload variant. +func TestGraphDriveFoldersAndResolution(t *testing.T) { + base := bootGraphService(t, 0) + + // ===== POST /v1.0/me/drive/root/children -> createFolder ===== + + folderBody, _ := json.Marshal(map[string]any{"name": "Backups", "folder": map[string]any{}}) + status, body := graphDo(t, "POST", base+"/v1.0/me/drive/root/children", graphToken, + folderBody, map[string]string{"Content-Type": "application/json"}) + if status != 201 { + t.Fatalf("createFolder -> status %d, want 201; body %s", status, body) + } + folder := graphJSON(t, body) + folderID, _ := folder["id"].(string) + if folderID == "" { + t.Fatalf("folder id missing: %s", body) + } + if _, ok := folder["folder"].(map[string]any); !ok { + t.Fatalf("created item lacks folder facet: %s", body) + } + + // Creating the same folder again defaults to fail -> 409. + status, _ = graphDo(t, "POST", base+"/v1.0/me/drive/root/children", graphToken, + folderBody, map[string]string{"Content-Type": "application/json"}) + if status != 409 { + t.Fatalf("duplicate createFolder -> status %d, want 409", status) + } + + // ===== GET /v1.0/me/drive/root:/Backups:/?select=id -> folder id ===== + + status, body = graphDo(t, "GET", base+"/v1.0/me/drive/root:/Backups:/?select=id", graphToken, nil, nil) + if status != 200 { + t.Fatalf("resolve folder -> status %d, want 200; body %s", status, body) + } + resolved := graphJSON(t, body) + if resolved["id"] != folderID { + t.Fatalf("resolved id = %v, want %v", resolved["id"], folderID) + } + + // Missing path -> 404. + status, _ = graphDo(t, "GET", base+"/v1.0/me/drive/root:/NoSuchFolder:/?select=id", graphToken, nil, nil) + if status != 404 { + t.Fatalf("resolve missing folder -> status %d, want 404", status) + } + + // ===== upload into the folder: PUT items/{parentId}:/{name}:/content ===== + + payload := []byte("folder-scoped-content") + status, body = graphDo(t, "PUT", + base+"/v1.0/me/drive/items/"+folderID+":/photo.bin:/content", graphToken, + payload, map[string]string{"Content-Type": "application/octet-stream"}) + if status != 201 { + t.Fatalf("folder upload -> status %d, want 201; body %s", status, body) + } + item := graphJSON(t, body) + itemID, _ := item["id"].(string) + parentRef, _ := item["parentReference"].(map[string]any) + if parentRef == nil || parentRef["id"] != folderID { + t.Fatalf("folder upload parentReference = %v, want id %v", item["parentReference"], folderID) + } + + // Unknown parent id -> 404. + status, _ = graphDo(t, "PUT", + base+"/v1.0/me/drive/items/folder-does-not-exist:/photo.bin:/content", graphToken, + payload, nil) + if status != 404 { + t.Fatalf("upload to unknown parent -> status %d, want 404", status) + } + + // ===== per-parent listing ===== + + // The folder's children contain photo.bin. + status, body = graphDo(t, "GET", base+"/v1.0/me/drive/items/"+folderID+"/children", graphToken, nil, nil) + if status != 200 { + t.Fatalf("list folder children -> status %d, want 200; body %s", status, body) + } + var children graphODataList + if err := json.Unmarshal(body, &children); err != nil { + t.Fatalf("unmarshal children: %v", err) + } + foundInFolder := false + for _, c := range children.Value { + if c["id"] == itemID { + foundInFolder = true + } + } + if !foundInFolder { + t.Fatalf("photo.bin not listed in its folder's children: %s", body) + } + + // Root children contain the folder but NOT the nested file. + status, body = graphDo(t, "GET", base+"/v1.0/me/drive/root/children", graphToken, nil, nil) + if status != 200 { + t.Fatalf("list root children -> status %d, want 200", status) + } + var rootChildren graphODataList + if err := json.Unmarshal(body, &rootChildren); err != nil { + t.Fatalf("unmarshal root children: %v", err) + } + rootHasFolder, rootHasNested := false, false + for _, c := range rootChildren.Value { + if c["id"] == folderID { + rootHasFolder = true + } + if c["id"] == itemID { + rootHasNested = true + } + } + if !rootHasFolder { + t.Fatal("created folder missing from root children") + } + if rootHasNested { + t.Fatal("nested file leaked into root children; listing must be per-parent") + } + + // createFolder inside a folder via items/{id}/children. + subBody, _ := json.Marshal(map[string]any{"name": "Nested", "folder": map[string]any{}}) + status, body = graphDo(t, "POST", base+"/v1.0/me/drive/items/"+folderID+"/children", graphToken, + subBody, map[string]string{"Content-Type": "application/json"}) + if status != 201 { + t.Fatalf("nested createFolder -> status %d, want 201; body %s", status, body) + } + sub := graphJSON(t, body) + subParent, _ := sub["parentReference"].(map[string]any) + if subParent == nil || subParent["id"] != folderID { + t.Fatalf("nested folder parentReference = %v, want id %v", sub["parentReference"], folderID) + } + + // ===== quota still answers a select=quota query ===== + + status, body = graphDo(t, "GET", base+"/v1.0/me/drive?select=quota", graphToken, nil, nil) + if status != 200 { + t.Fatalf("drive select=quota -> status %d, want 200", status) + } + drive := graphJSON(t, body) + if _, ok := drive["quota"].(map[string]any); !ok { + t.Fatalf("drive response lacks quota object: %s", body) + } +} + +// TestGraphDriveUploadSession proves the strict resumable protocol: +// sequential contiguous chunks, 416 on violations, 202 + nextExpectedRanges +// mid-session, 201 + driveItem on the final range, session gone afterwards. +func TestGraphDriveUploadSession(t *testing.T) { + base := bootGraphService(t, 0) + host := strings.TrimPrefix(base, "http://") + + original := allByteValues(2500) + total := len(original) + + // ===== POST createUploadSession ===== + + sessBody, _ := json.Marshal(map[string]any{ + "item": map[string]any{"@microsoft.graph.conflictBehavior": "rename", "name": "big.bin"}, + }) + status, body := graphDo(t, "POST", base+"/v1.0/me/drive/root:/big.bin:/createUploadSession", + graphToken, sessBody, map[string]string{"Content-Type": "application/json"}) + if status != 200 { + t.Fatalf("createUploadSession -> status %d, want 200; body %s", status, body) + } + sess := graphJSON(t, body) + uploadURL, _ := sess["uploadUrl"].(string) + if !strings.HasPrefix(uploadURL, "http://"+host+"/v1.0/_upload/") { + t.Fatalf("uploadUrl = %q, want prefix http://%s/v1.0/_upload/", uploadURL, host) + } + if _, ok := sess["expirationDateTime"].(string); !ok { + t.Fatalf("expirationDateTime missing: %s", body) + } + + putChunk := func(start, end int, declaredTotal int) (int, []byte) { + return graphDo(t, "PUT", uploadURL, "", original[start:end+1], map[string]string{ + "Content-Range": fmt.Sprintf("bytes %d-%d/%d", start, end, declaredTotal), + "Content-Type": "application/octet-stream", + }) + } + + // ===== chunk 1: bytes 0-999 -> 202 with nextExpectedRanges ["1000-"] ===== + + status, body = putChunk(0, 999, total) + if status != 202 { + t.Fatalf("chunk 1 -> status %d, want 202; body %s", status, body) + } + mid := graphJSON(t, body) + ranges, _ := mid["nextExpectedRanges"].([]any) + if len(ranges) != 1 || ranges[0] != "1000-" { + t.Fatalf("nextExpectedRanges = %v, want [\"1000-\"]", mid["nextExpectedRanges"]) + } + if _, ok := mid["expirationDateTime"].(string); !ok { + t.Fatalf("202 lacks expirationDateTime: %s", body) + } + + // ===== violations -> 416 ===== + + // Out-of-order: resending the first chunk when offset 1000 is expected. + status, _ = putChunk(0, 999, total) + if status != 416 { + t.Fatalf("out-of-order chunk -> status %d, want 416", status) + } + // Gap: skipping ahead. + status, _ = putChunk(2000, 2499, total) + if status != 416 { + t.Fatalf("gap chunk -> status %d, want 416", status) + } + // Inconsistent total across chunks. + status, _ = graphDo(t, "PUT", uploadURL, "", original[1000:2000], map[string]string{ + "Content-Range": fmt.Sprintf("bytes 1000-1999/%d", total+7), + }) + if status != 416 { + t.Fatalf("inconsistent total -> status %d, want 416", status) + } + // end < start. + status, _ = graphDo(t, "PUT", uploadURL, "", []byte("x"), map[string]string{ + "Content-Range": fmt.Sprintf("bytes 1000-999/%d", total), + }) + if status != 416 { + t.Fatalf("end < start -> status %d, want 416", status) + } + // Malformed header -> 400. + status, _ = graphDo(t, "PUT", uploadURL, "", []byte("x"), map[string]string{ + "Content-Range": "bytes nonsense", + }) + if status != 400 { + t.Fatalf("malformed Content-Range -> status %d, want 400", status) + } + + // ===== chunk 2: bytes 1000-1999 -> 202, next 2000 ===== + + status, body = putChunk(1000, 1999, total) + if status != 202 { + t.Fatalf("chunk 2 -> status %d, want 202; body %s", status, body) + } + mid = graphJSON(t, body) + ranges, _ = mid["nextExpectedRanges"].([]any) + if len(ranges) != 1 || ranges[0] != "2000-" { + t.Fatalf("nextExpectedRanges after chunk 2 = %v, want [\"2000-\"]", mid["nextExpectedRanges"]) + } + + // ===== final chunk: bytes 2000-2499 -> 201 + driveItem ===== + + status, body = putChunk(2000, total-1, total) + if status != 201 { + t.Fatalf("final chunk -> status %d, want 201; body %s", status, body) + } + item := graphJSON(t, body) + itemID, _ := item["id"].(string) + if itemID == "" { + t.Fatalf("final driveItem id missing: %s", body) + } + if item["name"] != "big.bin" { + t.Fatalf("final driveItem name = %v, want big.bin", item["name"]) + } + if int64(item["size"].(float64)) != int64(total) { + t.Fatalf("final driveItem size = %v, want %d", item["size"], total) + } + + // ===== chunks after completion are rejected ===== + + status, _ = putChunk(0, 999, total) + if status != 404 { + t.Fatalf("chunk after completion -> status %d, want 404 (session gone)", status) + } + + // ===== assembled content round-trips byte-exact ===== + + status, got := graphDo(t, "GET", base+"/v1.0/me/drive/items/"+itemID+"/content", graphToken, nil, nil) + if status != 200 { + t.Fatalf("get assembled content -> status %d, want 200", status) + } + if !bytes.Equal(got, original) { + t.Fatalf("assembled content not byte-equal: got %d bytes, want %d", len(got), len(original)) + } + + // ===== unknown session -> 404 ===== + + status, _ = graphDo(t, "PUT", base+"/v1.0/_upload/sess-does-not-exist", "", []byte("x"), map[string]string{ + "Content-Range": "bytes 0-0/1", + }) + if status != 404 { + t.Fatalf("unknown session -> status %d, want 404", status) + } +} + +// TestGraphDriveSessionInFolder proves the items/{parentId}:/{name}: session +// variant: the final driveItem lands in the folder. +func TestGraphDriveSessionInFolder(t *testing.T) { + base := bootGraphService(t, 0) + + folderBody, _ := json.Marshal(map[string]any{"name": "Uploads", "folder": map[string]any{}}) + status, body := graphDo(t, "POST", base+"/v1.0/me/drive/root/children", graphToken, + folderBody, map[string]string{"Content-Type": "application/json"}) + if status != 201 { + t.Fatalf("createFolder -> status %d, want 201; body %s", status, body) + } + folderID, _ := graphJSON(t, body)["id"].(string) + + status, body = graphDo(t, "POST", + base+"/v1.0/me/drive/items/"+folderID+":/video.bin:/createUploadSession", + graphToken, []byte("{}"), map[string]string{"Content-Type": "application/json"}) + if status != 200 { + t.Fatalf("createUploadSession in folder -> status %d, want 200; body %s", status, body) + } + uploadURL, _ := graphJSON(t, body)["uploadUrl"].(string) + if uploadURL == "" { + t.Fatalf("uploadUrl missing: %s", body) + } + + payload := []byte("single-chunk-session-payload") + status, body = graphDo(t, "PUT", uploadURL, "", payload, map[string]string{ + "Content-Range": fmt.Sprintf("bytes 0-%d/%d", len(payload)-1, len(payload)), + }) + if status != 201 { + t.Fatalf("single final chunk -> status %d, want 201; body %s", status, body) + } + item := graphJSON(t, body) + parentRef, _ := item["parentReference"].(map[string]any) + if parentRef == nil || parentRef["id"] != folderID { + t.Fatalf("session item parentReference = %v, want id %v", item["parentReference"], folderID) + } + + itemID, _ := item["id"].(string) + status, got := graphDo(t, "GET", base+"/v1.0/me/drive/items/"+itemID+"/content", graphToken, nil, nil) + if status != 200 || !bytes.Equal(got, payload) { + t.Fatalf("folder session content mismatch: status %d", status) + } +} + +// TestGraphDriveOversizeBody413 proves a small max_body_bytes turns oversize +// simple uploads into a 413 instead of silent truncation. +func TestGraphDriveOversizeBody413(t *testing.T) { + base := bootGraphService(t, 1024) + + status, _ := graphDo(t, "PUT", base+"/v1.0/me/drive/root:/big.bin:/content", graphToken, + bytes.Repeat([]byte{0x5A}, 2048), nil) + if status != http.StatusRequestEntityTooLarge { + t.Fatalf("oversize simple upload -> status %d, want 413", status) + } +} diff --git a/internal/engine/photos_media_plane_test.go b/internal/engine/photos_media_plane_test.go new file mode 100644 index 0000000..f2b26f4 --- /dev/null +++ b/internal/engine/photos_media_plane_test.go @@ -0,0 +1,328 @@ +package engine + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "path/filepath" + "strings" + "testing" + "time" + + "stuntapi.com/stunt/internal/manifest" +) + +// bootPhotosService boots the photos-style reference adapter on a free port +// and returns its base URL. +func bootPhotosService(t *testing.T) string { + t.Helper() + + adapterDir, err := filepath.Abs(filepath.Join("..", "..", "adapters", "photos-style")) + if err != nil { + t.Fatal(err) + } + stateDir := t.TempDir() + m := &manifest.Manifest{ + Path: filepath.Join(stateDir, "stunt.yaml"), + Version: 1, + Network: manifest.Network{Mode: "port", BasePort: 0}, + Services: map[string]manifest.Service{ + "photos": {Adapter: adapterDir}, + }, + } + e, err := New(m) + if err != nil { + t.Fatalf("engine.New: %v", err) + } + t.Cleanup(func() { e.Close() }) + addrs, cancel, err := e.ServeForTest(context.Background()) + if err != nil { + t.Fatalf("ServeForTest: %v", err) + } + t.Cleanup(cancel) + time.Sleep(50 * time.Millisecond) + return addrs["photos"] +} + +// photosUpload POSTs raw bytes to /v1/uploads and returns the uploadToken. +func photosUpload(t *testing.T, base, token string, payload []byte, contentType string) string { + t.Helper() + req, _ := http.NewRequest("POST", base+"/v1/uploads", bytes.NewReader(payload)) + req.Header.Set("Content-Type", contentType) + req.Header.Set("Authorization", "Bearer "+token) + req.Header.Set("X-Goog-Upload-Protocol", "raw") + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != 200 { + t.Fatalf("POST /v1/uploads -> status %d, want 200; body %s", resp.StatusCode, body) + } + uploadToken := strings.TrimSpace(string(body)) + if uploadToken == "" { + t.Fatal("empty uploadToken") + } + return uploadToken +} + +// photosBatchCreateOne creates one media item from an uploadToken and returns +// the created mediaItem object. +func photosBatchCreateOne(t *testing.T, base, token, uploadToken, fileName string) map[string]any { + t.Helper() + body, status := photosPostJSONAuth(t, base+"/v1/mediaItems:batchCreate", token, map[string]any{ + "newMediaItems": []any{ + map[string]any{ + "simpleMediaItem": map[string]any{ + "uploadToken": uploadToken, + "fileName": fileName, + }, + }, + }, + }) + if status != 200 { + t.Fatalf("batchCreate -> status %d, want 200; body %s", status, body) + } + var resp map[string]any + if err := json.Unmarshal([]byte(body), &resp); err != nil { + t.Fatalf("unmarshal batchCreate: %v", err) + } + results, ok := resp["newMediaItemResults"].([]any) + if !ok || len(results) != 1 { + t.Fatalf("newMediaItemResults = %v, want 1 result", resp["newMediaItemResults"]) + } + item, ok := results[0].(map[string]any)["mediaItem"].(map[string]any) + if !ok { + t.Fatalf("mediaItem missing in %v", results[0]) + } + return item +} + +// getRaw GETs a URL without auth and returns status, body, and content type. +func getRaw(t *testing.T, url string) (int, []byte, string) { + t.Helper() + resp, err := http.Get(url) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + return resp.StatusCode, body, resp.Header.Get("Content-Type") +} + +// TestPhotosMediaPlane proves the media plane is real: uploaded bytes are +// stored, linked by batchCreate, addressable through a read-time baseUrl +// derived from the request host, and served back byte-exact only when the +// Google download suffix (=d / =dv) is present. +func TestPhotosMediaPlane(t *testing.T) { + base := bootPhotosService(t) + host := strings.TrimPrefix(base, "http://") + + code := photosAuthorize(t, base, "http://localhost:8080/callback", "state-mp", "photos-test-client-id") + accessToken := photosExchange(t, base, code, "photos-test-client-id", "photos-test-client-secret", "http://localhost:8080/callback") + + // Byte-fidelity fixture: PNG magic + all 256 byte values. + original := allByteValues(1024) + + uploadToken := photosUpload(t, base, accessToken, original, "image/png") + item := photosBatchCreateOne(t, base, accessToken, uploadToken, "fixture.png") + mediaID, _ := item["id"].(string) + if mediaID == "" { + t.Fatal("created mediaItem has no id") + } + + // baseUrl must be computed from the request host at read time. + wantBaseURL := "http://" + host + "/v1/media-dl/" + mediaID + if item["baseUrl"] != wantBaseURL { + t.Fatalf("batchCreate baseUrl = %v, want %s", item["baseUrl"], wantBaseURL) + } + + // ===== GET /v1/mediaItems/{id} returns the item with read-time baseUrl ===== + + body, status := photosGetAuth(t, base+"/v1/mediaItems/"+mediaID, accessToken) + if status != 200 { + t.Fatalf("GET mediaItems/{id} -> status %d, want 200; body %s", status, body) + } + var got map[string]any + if err := json.Unmarshal([]byte(body), &got); err != nil { + t.Fatalf("unmarshal item: %v", err) + } + if got["id"] != mediaID { + t.Fatalf("item id = %v, want %s", got["id"], mediaID) + } + if got["baseUrl"] != wantBaseURL { + t.Fatalf("get baseUrl = %v, want %s", got["baseUrl"], wantBaseURL) + } + + // Unknown item id -> 404. + _, status = photosGetAuth(t, base+"/v1/mediaItems/mock-media-does-not-exist", accessToken) + if status != 404 { + t.Fatalf("GET unknown mediaItem -> status %d, want 404", status) + } + + // ===== media-dl strict =d semantics ===== + + // =d serves the ORIGINAL bytes exactly. + status, dlBody, ctype := getRaw(t, wantBaseURL+"=d") + if status != 200 { + t.Fatalf("GET baseUrl=d -> status %d, want 200", status) + } + if !bytes.Equal(dlBody, original) { + t.Fatalf("=d download is not byte-equal to the upload: got %d bytes, want %d", len(dlBody), len(original)) + } + if ctype != "image/png" { + t.Fatalf("=d content type = %q, want the recorded image/png", ctype) + } + + // =dv also serves the original bytes (video suffix). + status, dvBody, _ := getRaw(t, wantBaseURL+"=dv") + if status != 200 { + t.Fatalf("GET baseUrl=dv -> status %d, want 200", status) + } + if !bytes.Equal(dvBody, original) { + t.Fatal("=dv download is not byte-equal to the upload") + } + + // A bare baseUrl (no suffix) serves a DERIVATIVE: 200 but clearly + // different bytes, so a client that forgets =d fails byte-comparison. + status, derivBody, _ := getRaw(t, wantBaseURL) + if status != 200 { + t.Fatalf("GET bare baseUrl -> status %d, want 200", status) + } + if bytes.Equal(derivBody, original) { + t.Fatal("bare baseUrl served the original bytes; want a distinct derivative payload") + } + + // Unknown blob id -> 404, with and without the suffix. + status, _, _ = getRaw(t, base+"/v1/media-dl/mock-media-unknown=d") + if status != 404 { + t.Fatalf("media-dl unknown id =d -> status %d, want 404", status) + } + status, _, _ = getRaw(t, base+"/v1/media-dl/mock-media-unknown") + if status != 404 { + t.Fatalf("media-dl unknown id -> status %d, want 404", status) + } +} + +// TestPhotosPagination proves list and search honor pageSize/pageToken and +// emit nextPageToken while more items remain. +func TestPhotosPagination(t *testing.T) { + base := bootPhotosService(t) + + code := photosAuthorize(t, base, "http://localhost:8080/callback", "state-pg", "photos-test-client-id") + accessToken := photosExchange(t, base, code, "photos-test-client-id", "photos-test-client-secret", "http://localhost:8080/callback") + + const total = 5 + created := map[string]bool{} + for i := 0; i < total; i++ { + payload := []byte(fmt.Sprintf("photo-payload-%d-", i)) + tok := photosUpload(t, base, accessToken, payload, "image/jpeg") + item := photosBatchCreateOne(t, base, accessToken, tok, fmt.Sprintf("p%d.jpg", i)) + id, _ := item["id"].(string) + created[id] = true + } + + // ===== GET list pagination: pageSize=2 -> pages of 2, 2, 1 ===== + + seen := map[string]bool{} + pageToken := "" + pages := 0 + for { + url := base + "/v1/mediaItems?pageSize=2" + if pageToken != "" { + url += "&pageToken=" + pageToken + } + body, status := photosGetAuth(t, url, accessToken) + if status != 200 { + t.Fatalf("list page -> status %d, want 200; body %s", status, body) + } + var resp map[string]any + if err := json.Unmarshal([]byte(body), &resp); err != nil { + t.Fatalf("unmarshal list page: %v", err) + } + items, _ := resp["mediaItems"].([]any) + if len(items) == 0 { + t.Fatalf("list page %d has no items", pages) + } + if len(items) > 2 { + t.Fatalf("list page %d has %d items, want <= pageSize 2", pages, len(items)) + } + for _, it := range items { + id, _ := it.(map[string]any)["id"].(string) + if seen[id] { + t.Fatalf("item %s appeared on two pages", id) + } + seen[id] = true + } + pages++ + next, _ := resp["nextPageToken"].(string) + if next == "" { + break + } + pageToken = next + if pages > total { + t.Fatal("pagination did not terminate") + } + } + if pages != 3 { + t.Fatalf("list pagination took %d pages, want 3 (2+2+1)", pages) + } + if len(seen) != total { + t.Fatalf("list pagination returned %d distinct items, want %d", len(seen), total) + } + for id := range created { + if !seen[id] { + t.Fatalf("created item %s missing from paginated list", id) + } + } + + // ===== search pagination (pageSize/pageToken in the JSON body) ===== + + seen = map[string]bool{} + pageToken = "" + pages = 0 + for { + searchBody := map[string]any{"pageSize": 3} + if pageToken != "" { + searchBody["pageToken"] = pageToken + } + body, status := photosPostJSONAuth(t, base+"/v1/mediaItems:search", accessToken, searchBody) + if status != 200 { + t.Fatalf("search page -> status %d, want 200; body %s", status, body) + } + var resp map[string]any + if err := json.Unmarshal([]byte(body), &resp); err != nil { + t.Fatalf("unmarshal search page: %v", err) + } + items, _ := resp["mediaItems"].([]any) + if len(items) > 3 { + t.Fatalf("search page has %d items, want <= pageSize 3", len(items)) + } + for _, it := range items { + id, _ := it.(map[string]any)["id"].(string) + if seen[id] { + t.Fatalf("search item %s appeared on two pages", id) + } + seen[id] = true + } + pages++ + next, _ := resp["nextPageToken"].(string) + if next == "" { + break + } + pageToken = next + if pages > total { + t.Fatal("search pagination did not terminate") + } + } + if pages != 2 { + t.Fatalf("search pagination took %d pages, want 2 (3+2)", pages) + } + if len(seen) != total { + t.Fatalf("search pagination returned %d distinct items, want %d", len(seen), total) + } +} diff --git a/internal/engine/request_host_test.go b/internal/engine/request_host_test.go new file mode 100644 index 0000000..1d9e5d1 --- /dev/null +++ b/internal/engine/request_host_test.go @@ -0,0 +1,80 @@ +package engine + +import ( + "context" + "encoding/json" + "net/http" + "strings" + "testing" + "time" + + "stuntapi.com/stunt/internal/manifest" +) + +// hostAdapterYAML is a minimal adapter whose handler reports the request +// host as seen from Starlark via req["host"]. +const hostAdapterYAML = ` +id: hostecho +name: HostEcho +endpoints: + - route: /whoami + method: GET + handler: scripts/host.star#on_whoami +` + +const hostStar = ` +def on_whoami(req): + return respond(200, {"host": req["host"]}) +` + +// TestRequestHostVisibleToStarlark proves handlers can see the request Host +// header (r.Host) as req["host"], so adapters can mint self-referential URLs +// (photos baseUrl, upload session uploadUrl) that point back at the sim. +func TestRequestHostVisibleToStarlark(t *testing.T) { + adapterDir := t.TempDir() + writeFile(t, adapterDir, "adapter.yaml", hostAdapterYAML) + writeFile(t, adapterDir, "scripts/host.star", hostStar) + + stateDir := t.TempDir() + m := &manifest.Manifest{ + Path: stateDir + "/stunt.yaml", + Version: 1, + Network: manifest.Network{Mode: "port", BasePort: 0}, + Services: map[string]manifest.Service{ + "hostecho": {Adapter: adapterDir}, + }, + } + + e, err := New(m) + if err != nil { + t.Fatalf("engine.New: %v", err) + } + defer e.Close() + + addrs, cancel, err := e.ServeForTest(context.Background()) + if err != nil { + t.Fatalf("ServeForTest: %v", err) + } + defer cancel() + time.Sleep(50 * time.Millisecond) + + base := addrs["hostecho"] + wantHost := strings.TrimPrefix(base, "http://") + + resp, err := http.Get(base + "/whoami") + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + t.Fatalf("GET /whoami -> status %d, want 200", resp.StatusCode) + } + var out map[string]any + if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { + t.Fatalf("decode response: %v", err) + } + got, _ := out["host"].(string) + if got != wantHost { + t.Fatalf("req[\"host\"] = %q, want %q (the listen address)", got, wantHost) + } +} diff --git a/internal/engine/requestlog/recorder.go b/internal/engine/requestlog/recorder.go index 69d85a3..8e50bd9 100644 --- a/internal/engine/requestlog/recorder.go +++ b/internal/engine/requestlog/recorder.go @@ -45,14 +45,24 @@ func (r *Recorder) Wrap(next http.Handler) http.Handler { return } start := time.Now() - var reqBody []byte + // Capture the request body by teeing the stream rather than reading + // it up front: an up-front read through a LimitReader would TRUNCATE + // what downstream handlers see for bodies over the capture cap. The + // tee passes every byte through untouched and keeps at most + // maxBody+1 bytes for the log (the +1 marks truncation). + var capture *teeBody if req.Body != nil { - reqBody, _ = io.ReadAll(io.LimitReader(req.Body, maxBody+1)) - req.Body = io.NopCloser(bytes.NewReader(reqBody)) + capture = &teeBody{rc: req.Body} + req.Body = capture } rw := &capturingWriter{ResponseWriter: w, status: 200} next.ServeHTTP(rw, req) + var reqBody []byte + if capture != nil { + reqBody = capture.buf.Bytes() + } + dur := time.Since(start) e := Entry{ Seq: r.seq.Add(1), @@ -80,6 +90,28 @@ func capBody(b []byte) string { return string(b) } +// teeBody wraps a request body, passing every byte through to the reader +// while retaining the first maxBody+1 bytes for capture. It never alters +// what downstream handlers read. +type teeBody struct { + rc io.ReadCloser + buf bytes.Buffer +} + +func (t *teeBody) Read(p []byte) (int, error) { + n, err := t.rc.Read(p) + if n > 0 && t.buf.Len() <= maxBody { + keep := n + if room := maxBody + 1 - t.buf.Len(); keep > room { + keep = room + } + t.buf.Write(p[:keep]) + } + return n, err +} + +func (t *teeBody) Close() error { return t.rc.Close() } + // isWebSocketUpgrade reports whether the request is a WebSocket upgrade // (RFC 6455 §4.1). Such requests hijack the connection and must bypass // capture (mirrors engine.isWebSocketUpgrade, kept local to avoid a cycle). diff --git a/internal/manifest/manifest.go b/internal/manifest/manifest.go index 2bf0a50..4de29f0 100644 --- a/internal/manifest/manifest.go +++ b/internal/manifest/manifest.go @@ -54,6 +54,11 @@ type Service struct { Adapter string `yaml:"adapter,omitempty"` // adapter source spec (git:... or local path) or dir (optional) Rules []rules.Rule `yaml:"rules,omitempty"` Config map[string]any `yaml:"config,omitempty"` // optional per-service config (e.g. webhook_url) + + // MaxBodyBytes caps the request body size for this service. Bodies over + // the cap are rejected with HTTP 413 (never silently truncated). Zero + // means the engine default (1 MiB). + MaxBodyBytes int64 `yaml:"max_body_bytes,omitempty"` } type Manifest struct { diff --git a/internal/manifest/validate.go b/internal/manifest/validate.go index 209bcc7..3747678 100644 --- a/internal/manifest/validate.go +++ b/internal/manifest/validate.go @@ -63,6 +63,9 @@ func Validate(m *Manifest) error { if s.Adapter == "" && len(s.Rules) == 0 { return fmt.Errorf("manifest: service %q must have at least one of 'adapter' or 'rules'", n) } + if s.MaxBodyBytes < 0 { + return fmt.Errorf("manifest: service %q max_body_bytes must be >= 0 (0 = default 1 MiB)", n) + } // Only validate rules that exist. for i, r := range s.Rules { if r.Respond.Status == 0 && r.Respond.Behavior == "" && r.Respond.Body == nil { diff --git a/internal/starlark/vm.go b/internal/starlark/vm.go index 8068d72..8e71aec 100644 --- a/internal/starlark/vm.go +++ b/internal/starlark/vm.go @@ -18,6 +18,7 @@ const maxExecutionSteps = 1_000_000 type Request struct { Method string Path string + Host string // request host (r.Host) so handlers can mint self-referential URLs Headers map[string]string Body map[string]any RawBody string // raw request body as a string (for non-JSON bodies, e.g. binary uploads) @@ -161,6 +162,7 @@ func (vm *VM) Call(handlerName string, req Request) (Response, error) { reqVal, err := GoToStarlark(map[string]any{ "method": req.Method, "path": req.Path, + "host": req.Host, "headers": req.Headers, "body": req.Body, "raw_body": req.RawBody,