LLM Benchmark: sequential test for MongoDB

skills/typescript-client/SKILL.md

@@ -106,3 +106,10 @@ conn.db.user.onInsert((ctx, user) => console.log('Joined:', user.name));
 conn.db.user.onDelete((ctx, user) => console.log('Left:', user.name));
 conn.db.user.onUpdate((ctx, oldUser, newUser) => console.log('Updated:', newUser.name));
 ```
+
+## Gotchas
+
+- **`useTable` rows are `readonly`.** Copy before sorting/mutating, or it fails to type-check:
+  `const [rows] = useTable(tables.message); const sorted = [...rows].sort(...)`.
+- **bigint in JSX.** ids/counts from `t.u64()`/`t.i64()` columns are `bigint`, which React
+  cannot render. Wrap it: `{Number(row.id)}` or `{String(count)}`.

skills/typescript-server/SKILL.md

@@ -83,6 +83,14 @@ const spacetimedb = schema({ entity, record });  // ONE object, not spread args
 export default spacetimedb;
 ```
 
+The published module's **entry file must export the schema as default**. If you split tables
+(`schema.ts`) from reducers/lifecycle (`index.ts`), re-export it from the entry:
+
+```typescript
+// index.ts
+export { default } from './schema';   // re-export the schema for the module entry
+```
+
 ## Reducers
 
 Export name becomes the reducer name:
@@ -131,6 +139,10 @@ export const onDisconnect = spacetimedb.clientDisconnected((ctx) => { ... });
 // Auth: ctx.sender is the caller's Identity
 if (!row.owner.equals(ctx.sender)) throw new SenderError('unauthorized');
 
+// ctx.connectionId: the per-connection id, NULLABLE (ConnectionId | null) — null-check before use.
+// One Identity can hold several connections (multiple tabs/devices).
+if (ctx.connectionId) { /* ... */ }
+
 // Server timestamp (deterministic per reducer call)
 ctx.db.item.insert({ id: 0n, createdAt: ctx.timestamp });
 
@@ -161,6 +173,8 @@ export const tick = spacetimedb.reducer(
 
 // One-time: ScheduleAt.time(ctx.timestamp.microsSinceUnixEpoch + delayMicros)
 // Repeating: ScheduleAt.interval(60_000_000n)
+// Read time back from a scheduleAt value (tagged union):
+//   const micros = at.tag === 'time' ? at.value : at.value.microsSinceUnixEpoch;  // bigint
 ```
 
 ## Custom Types
@@ -183,6 +197,10 @@ const Shape = t.enum('Shape', {
 
 ## Views
 
+A client subscribing to a view receives only the rows it returns. Use a per-user view
+(keyed on `ctx.sender`) for per-viewer access control: deleting a row it depends on
+(e.g. a membership row) automatically drops the rows it was exposing from that client.
+
 ```typescript
 // Anonymous view (same for all clients):
 export const activeUsers = spacetimedb.anonymousView(

tools/llm-oneshot/apps/chat-app/prompts/base_mongodb.md

@@ -0,0 +1,37 @@
+# MongoDB Chat App - Base Prompt
+
+Create me a **real-time chat app** using **MongoDB as the backend**.
+
+Project root is:
+
+```
+apps/chat-app/
+```
+
+Create the project under a **timestamped folder**:
+
+```
+apps/chat-app/mongodb/chat-app-YYYYMMDD-HHMMSS/
+```
+
+Use `chat-app` as the **database name** for MongoDB.
+
+## Constraints
+
+- Work **entirely inside** your timestamped folder. Do not touch any other existing code.
+- Only create/modify code under:
+  - `apps/chat-app/mongodb/chat-app-YYYYMMDD-HHMMSS/server/` (server-side TypeScript)
+  - `apps/chat-app/mongodb/chat-app-YYYYMMDD-HHMMSS/client/` (client-side TypeScript/React)
+- Keep it minimal and readable.
+
+## UI Requirements
+
+- Dark theme with consistent color palette
+- Clear visual hierarchy — active states, hover effects, focus indicators
+- Responsive layout that works on desktop (mobile optional)
+- Loading and empty states for all data-dependent views
+- Visual feedback for user actions (button states, success/error indicators)
+
+## Features
+
+<!-- Include feature files below this line -->

tools/llm-oneshot/apps/chat-app/prompts/language/typescript-mongodb.md

@@ -0,0 +1,44 @@
+# Language: TypeScript + MongoDB
+
+Create this app using **MongoDB as the backend** with **TypeScript**.
+
+## Project Setup
+
+```
+apps/chat-app/staging/typescript/<LLM_MODEL>/mongodb/chat-app-YYYYMMDD-HHMMSS/
+```
+
+Database name: `chat-app`
+
+## Architecture
+
+**Backend:** Node.js + Express + Mongoose + Socket.io
+**Client:** React + Vite + TypeScript
+
+## Constraints
+
+- Only create/modify code under:
+  - `.../server/` (server-side TypeScript)
+  - `.../client/` (client-side TypeScript/React)
+- Keep it minimal and readable.
+
+## Branding & Styling
+
+- App title: **"MongoDB Chat"**
+- Dark theme using official MongoDB brand colors:
+  - Primary: `#00ED64` (MongoDB green)
+  - Primary hover: `#00C957` (darker green)
+  - Secondary: `#00684A` (MongoDB forest green)
+  - Background: `#001E2B` (MongoDB dark slate)
+  - Surface: `#023430` (deep green-slate)
+  - Border: `#1C2D38` (muted slate border)
+  - Text: `#E8EDEB` (light gray)
+  - Text muted: `#889397` (MongoDB gray)
+  - Accent: `#00ED64` (MongoDB green)
+  - Success: `#00ED64` (green for online indicators)
+  - Warning: `#FFC010` (MongoDB amber)
+  - Danger: `#FF4F4F` (MongoDB red)
+
+## Output
+
+Return only code blocks with file headers for the files you create.

tools/llm-sequential-upgrade/.gitattributes

@@ -0,0 +1,3 @@
+# Shell scripts here are run under bash (git-bash on Windows). Force LF so they
+# don't get CRLF-converted on checkout and break under stricter bash (WSL/CI).
+*.sh text eol=lf

tools/llm-sequential-upgrade/.gitignore

@@ -4,15 +4,12 @@
 **/results/**/.vite/
 **/results/**/drizzle/
 
+# Local env files inside generated apps (not committed)
+**/results/**/.env
+
 # Telemetry backup files
 **/telemetry/*.jsonl.bak
 
-
-# Playwright
-**/playwright/node_modules/
-**/playwright/test-results/
-**/playwright/playwright-report/
-
 # Isolation git repos inside generated apps (created by run.sh, cleaned up after)
 **/results/**/.git/
 # OTel collector live dump - not tracked
@@ -21,3 +18,13 @@ telemetry/metrics.jsonl
 
 # Raw telemetry contains PII (email, account IDs) - store privately
 **/telemetry/**/raw-telemetry.jsonl
+# Full Claude Code session transcript (large; contains absolute paths/PII) - store privately
+**/telemetry/**/session-transcript.jsonl
+# Verbose run transcripts (large, regenerable) - not tracked
+**/telemetry/**/run.log
+# Local absolute app paths (machine-specific)
+**/telemetry/**/app-dir.txt
+**/telemetry/**/metadata.json
+
+# Sequential-upgrade run output lives in the external spacetimedb-ai-test-results repo
+sequential-upgrade/sequential-upgrade-*/

tools/llm-sequential-upgrade/CLAUDE.md

@@ -1,8 +1,6 @@
-# Sequential Upgrade: LLM Cost-to-Done Benchmark
+# Chat App: Build Instructions
 
-You are running an automated benchmark that measures the **total cost to build a fully working chat app** — comparing SpacetimeDB vs PostgreSQL.
-
-Your job is to **generate, build, deploy, and fix** the app. Grading happens in a separate manual session — you do NOT test in the browser.
+Your job is to **generate, build, deploy, and fix** a fully working chat app. Verification happens in a separate session — you do NOT test in the browser.
 
 ---
 
@@ -30,10 +28,18 @@ Depending on the mode passed in the launch prompt:
 
 ---
 
+## Shell Syntax
+
+Windows host with both a Bash and a PowerShell tool — don't mix syntax. In the Bash tool use
+POSIX: `mkdir -p` not `New-Item`, `sleep` not `Start-Sleep`, `2>/dev/null` not `2>$null`,
+`VAR=x` not `$VAR=x`. PowerShell cmdlets in bash fail with "command not found".
+
+---
+
 ## Anti-Contamination
 
 Do NOT read any files under:
-- `../llm-oneshot/apps/chat-app/typescript/` (graded reference implementations)
+- `../llm-oneshot/apps/chat-app/typescript/` (reference implementations)
 - `../llm-oneshot/apps/chat-app/staging/`
 - Any other AI-generated app code in this workspace
 
@@ -46,7 +52,7 @@ Only read files you created, the backend instructions, and the feature prompts.
 1. Read `backends/<backend>.md` for pre-flight checks, phases, and deploy steps
 2. Read the language setup: `../llm-oneshot/apps/chat-app/prompts/language/typescript-<backend>.md`
 3. Read the feature prompt: `../llm-oneshot/apps/chat-app/prompts/composed/<NN>_<name>.md`
-4. Follow the phases in the backend file (generate backend → bindings → client → verify → deploy)
+4. Follow the phases in the backend file, in order
 5. Output `DEPLOY_COMPLETE` when the dev server is confirmed running
 
 For **upgrade**: only add the NEW features from the target level. Do not rewrite existing working features.
@@ -62,8 +68,6 @@ For **upgrade**: only add the NEW features from the target level. Do not rewrite
 5. Append to `ITERATION_LOG.md` (see format below)
 6. Output `FIX_COMPLETE`
 
-Do NOT do browser testing — that happens in the grading session.
-
 ---
 
 ## ITERATION_LOG.md
@@ -85,6 +89,6 @@ Append to this file after every fix. Never overwrite.
 
 ---
 
-## Cost Tracking
+## Telemetry
 
-Cost is tracked automatically via OpenTelemetry — do NOT estimate tokens or produce a COST_REPORT.md. That is generated automatically after the session ends.
+Do NOT estimate tokens or produce a COST_REPORT.md — that's captured automatically after the session ends.

tools/llm-sequential-upgrade/DEVELOP.md

@@ -263,20 +263,20 @@ llm-sequential-upgrade/
   DEVELOP.md                       # This file (for humans)
   run.sh                           # Code Agent launcher (generate/fix/upgrade)
   grade.sh                         # Grade Agent launcher (interactive Chrome MCP)
-  grade-playwright.sh              # Grade via Playwright (optional, deterministic)
+  templates/                       # BUG_REPORT.md / ITERATION_LOG.md formats
   docker-compose.otel.yaml         # OTel Collector container
   otel-collector-config.yaml       # Collector config (OTLP → JSON files)
   parse-telemetry.mjs              # Telemetry → COST_REPORT.md
   backends/
     spacetime.md                   # SpacetimeDB-specific phases
-    spacetime-sdk-rules.md         # SpacetimeDB SDK patterns
     spacetime-templates.md         # Code templates
+    # SDK reference = the official skills/typescript-{server,client}/SKILL.md
     postgres.md                    # PostgreSQL-specific phases
+    mongodb.md                     # MongoDB-specific phases
   test-plans/
     feature-01-basic-chat.md       # Per-feature browser test scripts
     ...
     feature-15-anonymous-migration.md
-    playwright/                    # Optional Playwright test suite
   telemetry/                       # Shared OTel Collector output
   sequential-upgrade/              # Sequential upgrade test variant
     sequential-upgrade-YYYYMMDD/   # Dated run with results, telemetry, inputs

tools/llm-sequential-upgrade/GRADING.md

@@ -11,6 +11,7 @@ You need TWO Chrome browser profiles so each user gets completely separate ident
 1. **Browser A (default profile):** Navigate to the app URL and register as "Alice"
    - SpacetimeDB: `http://localhost:6173`
    - PostgreSQL: `http://localhost:6273`
+   - MongoDB: `http://localhost:6373`
 
 2. **Switch to Browser B:** Use `switch_browser` to switch to the second Chrome profile
 

tools/llm-sequential-upgrade/GRADING_WORKFLOW.md

@@ -25,10 +25,12 @@ Code generation and fix iterations are token-tracked (the benchmark metric). Gra
 ```
 
 After generation, apps are running at:
-- **SpacetimeDB**: `http://localhost:5173` (run-index 0)
-- **PostgreSQL**: `http://localhost:5274` (run-index 1)
+- **SpacetimeDB**: `http://localhost:6173`
+- **PostgreSQL**: `http://localhost:6273`
+- **MongoDB**: `http://localhost:6373`
 
-Port offsets for parallel runs: run-index N uses ports `5173 + N*100` (spacetime) and `5174 + N*100` (postgres).
+Port offsets for parallel runs: run-index N adds N to the base port —
+`6173 + N` (spacetime), `6273 + N` (postgres), `6373 + N` (mongodb).
 
 ---
 

tools/llm-sequential-upgrade/README.md

@@ -21,14 +21,14 @@ Side-by-side results give a direct comparison of AI-generation cost across backe
 ## Directory contents
 
 - `run.sh`: orchestrates generation, upgrade, and fix sessions. Supports `--upgrade`, `--fix`, `--composed-prompt`, `--resume-session`.
-- `grade.sh` / `grade-agents.sh` / `grade-playwright.sh`: grading harnesses (manual + automated)
+- `grade.sh`: interactive grading harness (manual, Chrome MCP)
+- `templates/`: canonical `BUG_REPORT.md` / `ITERATION_LOG.md` formats for grading
 - `benchmark.sh` / `run-loop.sh`: batch runners for parallel or sequential benchmark execution
 - `cleanup.sh` / `reset-app.sh`: dev utilities
 - `benchmark-viewer.html`: local viewer for METRICS_DATA.json files (open in browser, drop JSON)
 - `generate-report.mjs`: aggregate per-session cost-summary.json into a markdown report
 - `parse-telemetry.mjs`: parse OTel log stream into per-session cost-summary.json
-- `parse-playwright-results.mjs`: convert Playwright JSON output to grading markdown
-- `docker-compose.otel.yaml` / `otel-collector-config.yaml`: OTel collector + PostgreSQL
+- `docker-compose.otel.yaml` / `otel-collector-config.yaml`: OTel collector + PostgreSQL + MongoDB
 - `backends/`: per-backend setup / SDK reference documents given to the AI
 - `perf-benchmark/`: runtime throughput benchmark (msgs/sec) for the AI-generated apps
 - `CLAUDE.md` / `DEVELOP.md` / `GRADING.md` / `GRADING_WORKFLOW.md`: process documentation