Import bounded batches of complete Rheon trajectories

.github/workflows/tests.yml

@@ -20,6 +20,8 @@ jobs:
       - run: python -m unittest discover -s tests
       - run: node --check static/workbench.js
       - run: node --check static/saved-selections.js
+      - run: node --check static/saved-searches.js
+      - run: node tests/test_saved_searches_controller.cjs
       - run: node --check static/curation.js
       - run: node --check static/bulk_import.js
       - run: node --check static/caption_import.js
@@ -36,6 +38,8 @@ jobs:
       - run: node --check static/meshes.js
       - run: node tests/test_meshes_controller.cjs
       - run: node --check static/sequences.js
+      - run: node --check static/sequence-batch.js
+      - run: node tests/test_sequence_batch_controller.cjs
       - run: node tests/test_curation_controller.cjs
       - run: node tests/test_browser_page_load.cjs
       - run: node tests/test_qa_artifacts.cjs
@@ -67,6 +71,9 @@ jobs:
       - run: node tests/browser_saved_selections.cjs
         env:
           BROWSER: /usr/bin/google-chrome
+      - run: node tests/browser_saved_searches.cjs
+        env:
+          BROWSER: /usr/bin/google-chrome
       - run: node tests/browser_saved_selection_intent.cjs
         env:
           BROWSER: /usr/bin/google-chrome
@@ -135,6 +142,9 @@ jobs:
       - run: node tests/browser_sequences.cjs
         env:
           BROWSER: /usr/bin/google-chrome
+      - run: node tests/browser_sequence_batch.cjs
+        env:
+          BROWSER: /usr/bin/google-chrome
       - run: node tests/browser_sequences_actual.cjs
         env:
           BROWSER: /usr/bin/google-chrome

README.md

@@ -188,8 +188,9 @@ In **Dataset workbench**, choose records with the checkboxes, enter a selection
 name, then **Save selected records as a new set**. A set contains fixed record IDs,
 annotation/source revisions, and source hashes. Filters and searches remain live
 collection controls; new matching records never join a saved set automatically.
-There is no dynamic saved-search feature in this slice. Names may repeat; each set
-has its own ID. Saving creates a separate set rather than replacing membership.
+Dynamic saved searches use a separate panel and never supply membership. Names may
+repeat; each set has its own ID. Saving creates a separate set rather than replacing
+membership.
 
 Choose a saved set and **Open fixed selection** to replace the current selection,
 including records outside the current filter. The current filter, release settings
@@ -225,6 +226,27 @@ currency rather than review or release eligibility. POST `.../ID/rename` accepts
 Stale mutations return HTTP 409 / `conflict`; a deleted set returns 404 / `unavailable`.
 For release preview, project saved items to `{id, revision, source_revision}`.
 
+## Saved dynamic searches
+
+**Saved searches · dynamic results** stores the entered search/filter values,
+even before **Search collection** is clicked. **Open dynamic search** explicitly
+applies them to the current collection, starting at page 1. New matching records
+can appear; selected record/answer/preference revisions, editor drafts, fixed sets,
+review decisions and history URLs remain separate. Reload lists saved searches
+without automatically opening any. Exact metadata values restore through JSON
+entry, preserving interior LF/CR/CRLF and literal backslashes. Search text is
+single-line. Names may repeat; criteria are immutable, so save changed filters as
+a new search. Rename/delete are revision-checked. Delete retains collection records.
+
+`GET/POST /api/workbench/searches` lists or creates searches. Create accepts only
+`name` (1–120 code points) and `criteria` with exactly `q`, `kind`, `task`, `review`,
+`sort`, `label`, `group`, `rights`, using the existing filter limits and semantics.
+`GET .../ID` returns explicit `mode: "dynamic"`, `schema_version: 1` metadata;
+`POST .../ID/rename` accepts `revision`/`name`, and `.../ID/delete` accepts `revision`.
+At most 100 searches and 32 KiB strict UTF-8 JSON per write. Saved searches contain
+no membership or assets and grant no release authority. Unknown write responses
+require list inspection before an explicit repeat; requests are not replayed.
+
 ## Selected-release preview
 
 Select records, choose an export format and split targets, then click **Preview selected release**. This summary describes exactly the saved selected revisions, independently of the current collection filter. It shows task/review/class-target counts, eligibility blockers, connected source families (including unselected and deleted relatives), requested versus achievable splits, and format/rights warnings. Whole source families remain indivisible; ratios are targets, not guarantees of exact quotas.
@@ -343,6 +365,37 @@ and a byte-exact frozen release. It never runs the exporter or simulation.
 No models, downloads or training are needed for these sequence checks.
 Focused Python check: `python3 -m unittest discover -s tests -p test_sequences.py`.
 
+The local follow-on **Import a folder of Rheon trajectories** accepts one selected
+folder containing only `<trajectory-label>/run.json` and `frames.jsonl` pairs.
+Preflight checks all pairs before reading: at most 32 pairs / 64 files / 40 MiB
+total raw files, 64 KiB per manifest and 2 MiB per frames file. Each server request
+is independently capped at 3 MiB and uses the unchanged fee7b4a validator. These
+selected-file limits do not impose a server-wide quota across independent imports.
+The existing 40 MiB immutable selection/release budget includes ZIP overhead.
+
+Each valid pair commits one whole draft trajectory with initial history and a
+computed acquisition marker/hash receipt. Invalid or duplicate items are rejected
+without modifying existing records; earlier successful items remain. Stop prevents
+the next admission, while an in-flight item may complete. Unknown responses or
+storage failures pause without automatic replay. **Check pending trajectory** is
+read-only and matches the marker, declared labels/index and both consumed-file
+hashes. It confirms admission only, never current source integrity, rights or
+human review. Missing lookup does not prove cessation. Confirmation does not
+resume later items; dismissal/departure/reload makes no rollback claim. Progress
+is held in this page; inspect the collection after reload. Imports retain the
+independent editor and exact selected revision pairs.
+
+Folder labels remain declared context and do not automatically group independent
+families. Use the optional shared protected group only when runs must stay together.
+Review, rights/history, fixed saved selections and canonical whole-trajectory
+export use their existing controls. No time flattening, frame split or scientific
+training qualification is added. `tests/test_sequence_batch.py` and
+`tests/browser_sequence_batch.cjs` use the retained actual fee7b4a pair and separately
+labeled source-derived synthetic controls; they execute no simulation.
+Rheon's still-draft PR20 has advanced to `3bf61ba85d066cadb95cddffb29a2b40fe497ccf`
+with a Cargo artifact repair and a new allowed build-command form. This adapter
+does not adopt that contract form; it is rejected until a separately reviewed repin.
+
 Source is hosted at [MrScripty/Tuldok](https://github.com/MrScripty/Tuldok). A distribution license has not yet been selected.
 
 ### Import an annotated caption corpus

app.py

@@ -11,6 +11,7 @@
 import dataset_recipes
 import dataset_releases
 import saved_selections
+import saved_searches
 import grounded_candidates
 import caption_proposals
 import text_classification_proposals
@@ -19,6 +20,7 @@
 import caption_import
 import native_text_import
 import rheon_sequences
+import sequence_batch_import
 import meshes
 import base64
 import hashlib
@@ -148,6 +150,7 @@ def __init__(self, path):
         self.workbench = workbench.Workbench(self)
         self.releases = dataset_releases.Releases(self.workbench)
         self.selections = saved_selections.SavedSelections(self.workbench)
+        self.searches = saved_searches.SavedSearches(self.workbench)
         self.caption_imports = caption_import.CaptionImports(self.workbench)
         self.native_text_imports = native_text_import.NativeTextImports(self.workbench)
         self.grounded = grounded_candidates.Proposals(self.workbench)
@@ -387,10 +390,16 @@ def do_GET(self):
             try:
                 if path == '/api/workbench/selections':
                     return self.reply(dataset.selections.list())
+                if path == '/api/workbench/searches':
+                    return self.reply(dataset.searches.list())
+                if path.startswith('/api/workbench/searches/'):
+                    return self.reply(dataset.searches.get(path.removeprefix('/api/workbench/searches/')))
                 if path.startswith('/api/workbench/selections/'):
                     return self.reply(dataset.selections.load(path.removeprefix('/api/workbench/selections/')))
                 if path.startswith('/api/workbench/import-result/'):
                     return self.reply(bulk_import.find_result(dataset.workbench, path.rsplit('/', 1)[-1]))
+                if path.startswith('/api/workbench/sequence-import-result/'):
+                    return self.reply(sequence_batch_import.find_result(dataset.workbench, path.rsplit('/', 1)[-1]))
                 if path == '/api/workbench/text-classification-proposals':
                     return self.reply(dataset.text_classification_proposals.snapshot())
                 if path.startswith('/api/workbench/text-classification-proposals/'):
@@ -465,7 +474,7 @@ def do_GET(self):
                         self.end_headers()
                         shutil.copyfileobj(archive, self.wfile)
                     return
-                assets = {'/pumas-typed.js': ('pumas-typed.js', 'text/javascript'), '/pumas-gateways.js': ('pumas-gateways.js', 'text/javascript'), '/meshes.js': ('meshes.js', 'text/javascript'), '/sequences.js': ('sequences.js', 'text/javascript'), '/text-classification-proposals.js': ('text-classification-proposals.js', 'text/javascript'), '/caption-proposals.js': ('caption-proposals.js', 'text/javascript'), '/curation.js': ('curation.js', 'text/javascript'), '/caption-import.js': ('caption_import.js', 'text/javascript'), '/bulk-import.js': ('bulk_import.js', 'text/javascript'), '/saved-selections.js': ('saved-selections.js', 'text/javascript'), '/workbench': ('workbench.html', 'text/html'), '/workbench.js': ('workbench.js', 'text/javascript'), '/workbench.css': ('workbench.css', 'text/css'), '/': ('index.html', 'text/html'), '/app.js': ('app.js', 'text/javascript'), '/style.css': ('style.css', 'text/css'), '/rights-note.js': ('rights-note.js', 'text/javascript'), '/preferences.js': ('preferences.js', 'text/javascript'), '/instruction-responses.js': ('instruction-responses.js', 'text/javascript'), '/native-text-import.js': ('native_text_import.js', 'text/javascript')}
+                assets = {'/saved-searches.js': ('saved-searches.js', 'text/javascript'), '/sequence-batch.js': ('sequence-batch.js', 'text/javascript'), '/pumas-typed.js': ('pumas-typed.js', 'text/javascript'), '/pumas-gateways.js': ('pumas-gateways.js', 'text/javascript'), '/meshes.js': ('meshes.js', 'text/javascript'), '/sequences.js': ('sequences.js', 'text/javascript'), '/text-classification-proposals.js': ('text-classification-proposals.js', 'text/javascript'), '/caption-proposals.js': ('caption-proposals.js', 'text/javascript'), '/curation.js': ('curation.js', 'text/javascript'), '/caption-import.js': ('caption_import.js', 'text/javascript'), '/bulk-import.js': ('bulk_import.js', 'text/javascript'), '/saved-selections.js': ('saved-selections.js', 'text/javascript'), '/workbench': ('workbench.html', 'text/html'), '/workbench.js': ('workbench.js', 'text/javascript'), '/workbench.css': ('workbench.css', 'text/css'), '/': ('index.html', 'text/html'), '/app.js': ('app.js', 'text/javascript'), '/style.css': ('style.css', 'text/css'), '/rights-note.js': ('rights-note.js', 'text/javascript'), '/preferences.js': ('preferences.js', 'text/javascript'), '/instruction-responses.js': ('instruction-responses.js', 'text/javascript'), '/native-text-import.js': ('native_text_import.js', 'text/javascript')}
                 if path in assets:
                     name, kind = assets[path]
                     return self.reply((ROOT / 'static' / name).read_bytes(), content_type=kind + '; charset=utf-8')
@@ -486,16 +495,23 @@ def do_POST(self):
                     raise ValueError('Send JSON.')
                 length = int(self.headers.get('Content-Length', '0'))
                 path = urlsplit(self.path).path
-                limit = meshes.MAX_REQUEST if path == '/api/workbench/mesh-import' else rheon_sequences.MAX_REQUEST if path == '/api/workbench/sequence-import' else MAX_BODY
+                limit = saved_searches.MAX_REQUEST if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else meshes.MAX_REQUEST if path == '/api/workbench/mesh-import' else rheon_sequences.MAX_REQUEST if path in ('/api/workbench/sequence-import', '/api/workbench/sequence-import-item') else MAX_BODY
                 if not 0 < length <= limit:
                     raise ValueError('Request is too large or empty.')
                 raw_body = self.rfile.read(length)
-                body = meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
+                body = saved_searches.parse_request(raw_body) if path == '/api/workbench/searches' or path.startswith('/api/workbench/searches/') else sequence_batch_import.parse_request(raw_body) if path == '/api/workbench/sequence-import-item' else meshes.parse_json(raw_body) if path == '/api/workbench/mesh-import' else json.loads(raw_body)
                 if not isinstance(body, dict):
                     raise ValueError('Expected an object.')
                 path = urlsplit(self.path).path
                 if path == '/api/workbench/selections':
                     return self.reply(dataset.selections.create(body), 201)
+                if path == '/api/workbench/searches':
+                    return self.reply(dataset.searches.create(body), 201)
+                if path.startswith('/api/workbench/searches/'):
+                    parts = path.removeprefix('/api/workbench/searches/').split('/')
+                    if len(parts) != 2:
+                        raise workbench.WorkbenchError('Invalid saved search route.')
+                    return self.reply(dataset.searches.mutate(parts[0], parts[1], body))
                 if path.startswith('/api/workbench/selections/'):
                     parts = path.removeprefix('/api/workbench/selections/').split('/')
                     if len(parts) != 2:
@@ -525,6 +541,8 @@ def do_POST(self):
                     return self.reply(meshes.admit(dataset.workbench, body), 201)
                 if path == '/api/workbench/sequence-import':
                     return self.reply(rheon_sequences.admit(dataset.workbench, body), 201)
+                if path == '/api/workbench/sequence-import-item':
+                    return self.reply(sequence_batch_import.import_item(dataset.workbench, body), 201)
                 if path == '/api/workbench/curation':
                     return self.reply(curation.inspect(dataset.workbench, body))
                 if path == '/api/workbench/import-row':

docs/plans/bulk-rheon-sequences/design-review.md

@@ -0,0 +1,65 @@
+# Independent bulk Rheon design review
+
+Reviewed the plan against base `c93aee448e60b310eb5f9d926fa4185130e91a6d`, including
+`rheon_sequences.py`, `sequence_assets.py`, `immutable_assets.py`, Workbench save
+and history ownership, HTTP admission/error handling, and the existing collection,
+single-sequence and raw-bulk controllers. Read-only review; no implementation,
+simulation, provider execution or public mutation. No applicable local AGENTS or
+skills were found.
+
+**Admitted for implementation with the requirements below.** No fundamental
+composition or lifecycle blocker was found. The small SequenceAssets extension
+can place trusted acquisition context in its existing origin before immutable
+asset/record/history publication. Workbench human sequence review preserves that
+origin and enforces trajectory/initial-family groups. Collection `refresh()`
+does not replace the editor or mutate the selected-record map.
+
+1. Store the context only at `provenance.sequence_acquisition`; retain the existing
+   simulation origin and adapter hashes. Check marker uniqueness and call admission
+   under the same reentrant Dataset lock. Put the marker in the initial origin,
+   never a post-admission update. Validation may happen before that lock. Injected
+   asset/record/history failures must roll back the marker too. Classify storage
+   failures as uncertain/server failures; the existing HTTP fallback can otherwise
+   turn an OSError into a 400 and incorrectly permit continued scheduling.
+2. Lookup must require sequence kind and exact `tuldok_rheon_batch_v1` format,
+   fetch at most two matches to detect ambiguity, and return bounded admission
+   identity/context/hash evidence without loading a bundle or arrays. Do not reuse
+   raw `bulk_import.find_result`, whose namespace and row assumptions differ.
+   Neither lookup nor its receipt certifies current source integrity or review.
+   Client confirmation must bind marker, exact declared labels/index and both
+   hashes of the bytes actually encoded for the POST; mismatches remain uncertain.
+3. Preflight the entire selection before any contents read: one common root,
+   exact three-component relative paths, complete unique pairs, exact bounded
+   labels, file counts and positive per-file/aggregate sizes. Snapshot all files
+   and controls before awaits. Read each file once and retain only the current
+   pair's encoded/raw working data. The 32-pair/40 MiB selected-raw limits are
+   browser admission bounds, not a server-wide quota across independent POSTs.
+   Preserve the independent 40 MiB immutable-bundle selection/export budget,
+   including ZIP overhead. The new HTTP route needs its explicit 3 MiB cap and
+   closed UTF-8 JSON envelope, rejecting duplicate fields and invalid constants.
+4. Stop checks must follow asynchronous reads/hashes and precede every POST.
+   Pagehide must stop future scheduling and fence stale read, POST, lookup and
+   final-refresh continuations; it cannot undo an in-flight commit. Prevent
+   repeated controls/held responses from acting on a newer batch. A missing GET
+   result is inconclusive and keeps the pending item; confirmation never resumes
+   later pairs. Reload recovery is explicitly not durable. Dismissal makes no
+   rollback claim. Keep acquisition outcomes separate from a failed collection
+   refresh so a known committed receipt is not reclassified as a rejected item.
+5. Keep new draft state, permission notes, parents, human review, fixed selection
+   and whole-family export with their existing owners. Labels remain inert
+   declared context; only an explicit user group links otherwise unrelated runs.
+   Tests must retain dirty editor contents and exact selection snapshots, not
+   merely check that their panels remain visible.
+
+The planned SQLite/reopen, malformed producer fixture, race/failure, controller
+and actual selected-file Chromium coverage is appropriate. Add explicit checks
+for simultaneous marker collision, a marker borrowed from the raw-import
+namespace, pagehide during a read/held POST/lookup, and refresh failure after a
+valid creation receipt. Independent implementation review and exact-head full
+qualification remain required before completion.
+
+The producer PR20 head has advanced to
+`3bf61ba85d066cadb95cddffb29a2b40fe497ccf` according to the parent's fresh supported
+read. This slice deliberately keeps the admitted exporter/validator and retained
+actual fixture at `fee7b4a139574f87b259796b1ba8698a41d31ac1`. No newer contract is
+accepted by this review; adoption requires separate root-coordinated review.