Rebuild series ingest_locations on SyncFiles=false download (#302)

[stevevanhooser]

Closes #302. Ports VH-Lab/NDI-matlab#988 -- the SyncFiles=false half of the file-series cloud round trip.

What changes

ndi.cloud.filehandler.updateFileInfoForRemoteFiles now teaches series to survive a SyncFiles=false cloud round trip. DID strips ingest_locations before storing a document, so one that comes back declares n_present members with no location for any of them -- exactly the shape DID's MembersNotLocatable guard (DID-matlab#185 / DID-python's _reject_series_without_ingest_locations) refuses on add_docs. The whole SyncFiles=false path was blocked on that guard for any document carrying a populated series.

The manifest bytes are not on disk on this path -- SyncFiles=false does not download them -- so the reconstruction has to fetch. It goes through the DID custom_file_handler contract (VH-Lab/DID-matlab#201, VH-Lab/DID-python#88), the same contract DID uses on the read side, so a caller with a handler already can pass it in and both sides use the same one. With no handler, fetch_cloud_file is called directly, matching what the read-side handler does when a manifest is asked for by uid.

Manifests fetched here do not land in the DID file cache; a subsequent member open trips DID#201's lazy fetch, which is what populates the cache. Cache eviction on a later session is answered by the same re-fetch path.

Code

• src/ndi/cloud/filehandler.py:

  • updateFileInfoForRemoteFiles gains keyword-only custom_file_handler=None and client=None. After the file_info rewrite it checks _needs_series_reconstruction; when a series has n_present > 0 and empty ingest_locations, _reconstruct_series_from_cloud fetches each qualifying manifest to a scratch tempfile.TemporaryDirectory and calls reconstructSeriesIngestLocations before the temp dir is cleaned up.
  • _fetch_manifest dispatches through the handler when given (with seriesName="" in the context -- a non-empty seriesName marks a MEMBER fetch, where the handler treats ctx.uid as the file to retrieve and source_path as the manifest, the exact opposite of what we want here) or falls back to fetch_cloud_file.
  • _dispatch_custom_file_handler mirrors DID-python's arity-aware dispatch, so a two-argument handler still works.
  • A per-manifest fetch failure is logged; the entry is left un-reconstructed so DID's guard fires on add_docs with the document's own identity.

• Bridge yaml: updateFileInfoForRemoteFiles's matlab_last_sync_hash bumped to aea0c72c (the NDI-matlab main head that carries #988). custom_file_handler and client recorded as keyword-only arguments.

Tests

tests/test_cloud_series_reconstruction_remote.py (new, seven tests, all offline):

• TestFileInfoReshape -- every file_info location becomes ndicloud, ingest=0, delete_original=0, uid survives, location is ndic://<dataset>/<uid>. Covers both plain and series documents.
• TestReconstructionSkipsWhenNotNeeded -- a doc whose ingest_locations is already populated skips reconstruction (no fetch, no handler call). Same for a doc without series_info.
• TestReconstructionThroughAMockHandler -- end-to-end offline: a mock handler serves manifest bytes by uid; the test walks a doc through DID's database_add / database_search cycle so ingest_locations gets stripped (the exact shape a cloud round trip returns), reshapes via updateFileInfoForRemoteFiles(..., custom_file_handler=mock), and pins:
  • one entry per present member, ndic:// location, ingest=0.
  • exactly one manifest fetch total (a 28,000-member series must not provoke 28,000 fetches).
  • seriesName="" in the handler context on the manifest fetch.
  • a handler failure leaves ingest_locations empty so DID's guard fires on add.
  • a two-argument handler still works through the arity-aware dispatch.

Mirrors NDI-matlab's tests/+ndi/+unittest/+database/TestUpdateFileInfoForRemoteFilesShape.m (VH-Lab/NDI-matlab#987) shape-for-shape.

Verification

pytest tests/test_cloud_series_reconstruction.py \
       tests/test_cloud_series_reconstruction_remote.py \
       tests/test_cloud_download_ordering.py \
       tests/test_cloud_filehandler.py \
       tests/test_cloud_file_uid_enumeration.py

74 passed, 2 skipped. Bridge checks (test_matlab_bridge_hashes.py, test_matlab_bridge_completeness.py) all green with the local NDI-matlab checkout.

Unrelated GUI drift acknowledged

GEFManager's matlab_last_sync_hash bumped from 06f31a4e5 to 59c96e57 (VH-Lab/NDI-matlab#984 -- raise the open GEF Manager for a session instead of building a second one). MATLAB App Designer window uniqueness only, nothing to port on this side; hash bumped with a decision_log note so the drift check no longer trips on any PR opened after that commit landed.

Not in scope

A Python analog of NDI-matlab's FileSeriesRoundTripTest.testMembersSurviveADownloadFromTheCloudWithSyncFilesFalse -- the live-cloud round-trip test. Belongs in its own change, layered after this one; flagged in the closing note on #300 and to be filed as its own issue.

References

• Retire the manifest cloud-reference workaround and add an offline SyncFiles=false round-trip test once DID-matlab#201 lands VH-Lab/NDI-matlab#986 (parent tracking issue)
• Offline test for a cloud-only series manifest (SyncFiles=false shape) (#986) VH-Lab/NDI-matlab#987, Cloud companion: SyncFiles=false round-trip in FileSeriesRoundTripTest (#986) VH-Lab/NDI-matlab#988 (the MATLAB port)
• File series: accessors, remote ingest, an unlocatable-series guard, and one fetch per uid VH-Lab/DID-matlab#185 (the guard this fix satisfies)
• localSeriesManifestPath cannot fetch a remote manifest — series reads fail on cloud-only downloads VH-Lab/DID-matlab#201, Fetch a remote-only series manifest through the custom_file_handler (#87) VH-Lab/DID-python#88 (the customFileHandler contract for a manifest fetch)
• Retire the manifest cloud-reference workaround (companion to VH-Lab/NDI-matlab#986) #300 (parent, closed)

🤖 Generated with Claude Code

https://claude.ai/code/session_01AdGZ8fRPeY9JSpUHgXm9ao

---

Generated by Claude Code