Repository navigation
Retire the manifest cloud-reference workaround (companion to VH-Lab/NDI-matlab#986) #300
Description
Companion to VH-Lab/NDI-matlab#986. Ports the workaround retirement now that VH-Lab/DID-python#88 has landed the reads that make it unnecessary.
What the workaround is
ndi.cloud.filehandler.updateFileInfoForLocalFiles rewrites a downloaded document's file_info to point at the local files that came down with it. For a series MANIFEST it did two things a plain file does not:
- Set the local
filelocation (same as any file). - Append a second
ndicloudlocationndic://<cloudDatasetId>/<manifestUid>alongside.
The second location was there to give DID's custom_file_handler a non-file source path for member fetches. Without it, download_file_from_cloud was handed a plain local path and returned without producing anything, so every member of a downloaded series read as absent -- silently, because DID reads a handler that fails as a plain miss. See VH-Lab/NDI-matlab#966.
Why it can go now
- Offer a local-file manifest to the handler too (unblocks #188 for its main caller) VH-Lab/DID-matlab#191 taught the read path to reach a local-file manifest directly (a manifest whose only location is a local file is offered to the handler, and the guard against a handler that returns the manifest's own bytes makes that harmless).
- localSeriesManifestPath cannot fetch a remote manifest — series reads fail on cloud-only downloads VH-Lab/DID-matlab#201 closed the last case: a cloud-only manifest fetched through the same handler.
- Both landed. Both were ported to Fetch a remote-only series manifest through the custom_file_handler (#87) VH-Lab/DID-python#88 (merged).
That makes the second location dead weight -- DID never consults it anymore.
What to change
src/ndi/cloud/filehandler.py:
- In
updateFileInfoForLocalFiles, drop the block that appends_ndic_location_record(cloud_dataset_id, uid)to a manifest'slocations. Simplify the write-back around it (no more grown-list case). - Remove the now-unused helpers
_ndic_location_recordand_declared_series_names. - Update the docstring:
cloud_dataset_idstill drives the ingest_locations rebuild viareconstructSeriesIngestLocations(Cloud-downloaded documents refused by DID's MembersNotLocatable guard VH-Lab/NDI-matlab#958), but no longer the manifest's second location. - Comment update at the call site in
src/ndi/cloud/orchestration.py.
tests/test_cloud_series_reconstruction.py:
- Remove
TestTheManifestKeepsItsCloudReference(analog ofTestDownloadedSeriesManifestKeepsItsCloudLocation.m; its whole subject was pinning the second location). - Remove
test_a_member_of_a_downloaded_series_can_be_read-- it asserted the handler receives anndic://URI built from the manifest's second location, which no longer exists. End-to-end coverage of a member of a downloaded series lives on the cloud side, mirroring NDI-matlab'stestMembersSurviveADownloadFromTheCloud. - Add
TestSeriesManifestHasExactlyOneLocationas the regression guard: exactly one location, of typefile. This mirrorsTestUpdateFileInfoPreservesSeries.testSeriesManifestHasExactlyOneLocationadded in Retire the manifest cloud-reference workaround (#986) VH-Lab/NDI-matlab#989.
Bridge yamls:
- Bump
matlab_last_sync_hashonupdateFileInfoForLocalFilesto the NDI-matlab#989 head (2ad46c0f) and drop the "second location beside the local copy" language from the decision_log. - Companion notes in
src/ndi/cloud/sync/ndi_matlab_python_bridge.yaml.
Related work, deliberately out of scope
VH-Lab/NDI-matlab#988 also taught the SyncFiles=false path (updateFileInfoForRemoteFiles) to rebuild series ingest_locations by fetching the manifest bytes through the DID customFileHandler contract. NDI-python's updateFileInfoForRemoteFiles does not yet do this -- a document with a populated series returned from a SyncFiles=false cloud round trip will hit DID's MembersNotLocatable guard on the following add_docs. Separate issue, separate change.
References
- Retire the manifest cloud-reference workaround and add an offline SyncFiles=false round-trip test once DID-matlab#201 lands VH-Lab/NDI-matlab#986 (parent, MATLAB side)
- Retire the manifest cloud-reference workaround (#986) VH-Lab/NDI-matlab#989 (the MATLAB retirement PR)
- No test reads a file series MEMBER back out of the cloud: the round trip is only proven for the manifest VH-Lab/NDI-matlab#966 (why the workaround existed)
- Cloud-downloaded documents refused by DID's MembersNotLocatable guard VH-Lab/NDI-matlab#958 (why cloud_dataset_id stays in the signature)
- Offer a local-file manifest to the handler too (unblocks #188 for its main caller) VH-Lab/DID-matlab#191, localSeriesManifestPath cannot fetch a remote manifest — series reads fail on cloud-only downloads VH-Lab/DID-matlab#201 (DID-side changes that make the workaround unnecessary)
- Fetch a remote-only series manifest through the custom_file_handler (#87) VH-Lab/DID-python#88 (DID-python port that closed the loop)
Activity
stevevanhooser commentedon Sep 14, 2026
Closing as not planned. The MATLAB-side retirement was retracted (VH-Lab/NDI-matlab#989 closed unmerged, VH-Lab/NDI-matlab#986 closed as not planned), and the same premise fails on the Python side.
Why
The premise for retiring the workaround was that DID (post DID-matlab#191 / DID-python#88) could reach a manifest's local
filelocation on a member fetch and hand it to the customFileHandler. It can -- but "offering" asourcePathis not "servicing" one. DID iterates every manifest location and offers the local-file one to the handler; the NDI handler (in both languages) refuses anysourcePaththat does not start withndic://. Without the secondndicloudlocation on the manifest, the handler has nothing to service and every member of a downloaded series (SyncFiles=true) reads as absent -- silently, because DID reads a handler that produces nothing as a plain miss.Cloud CI on NDI-matlab#989 caught this on
FileSeriesRoundTripTest.testMembersSurviveADownloadFromTheCloud(SyncFiles=true) and.testTheBatchScopeIsTheSeriesAndNotTheWholeDocument, both failing withDID:SQLITEDB:open,chunkdata.bin_1 ... cannot be accessed.The NDI-python analog
ndi.cloud.filehandler.download_file_from_cloud(src/ndi/cloud/filehandler.py:677-678) has the same behavior:so the regression would have hit here too.
Full analysis on the NDI-matlab side:
State of the world
origin/mainstill carriesmatlab_last_sync_hash: "504c5bcf4"forupdateFileInfoForLocalFilesin both bridge YAMLs, so nothing needs reverting.updateFileInfoForLocalFilesstill appends the secondndicloudlocation on a series manifest,_ndic_location_recordand_declared_series_namesstay, andtests/test_cloud_series_reconstruction.py::TestThroughARealDatabase::test_a_member_of_a_downloaded_series_can_be_readremains as the guard against removing the workaround silently.Coverage note (out-of-scope item from this issue)
This issue's "Related work, deliberately out of scope" section already recorded that NDI-python's
updateFileInfoForRemoteFilesdoes not rebuild seriesingest_locationson the SyncFiles=false path via the DID customFileHandler contract (NDI-matlab#988's addition). Retracting this retirement does not change that gap, and there is no NDI-python test analogous to a SyncFiles=false / cloud-only-manifest read path (offline + cloud) matching NDI-matlab#987/#988. That gap is worth its own tighter issue when it is picked up, and is not being folded back into this closed issue.Similarly, an integration test analogous to NDI-matlab's
FileSeriesRoundTripTest.testMembersSurviveADownloadFromTheCloudon the live cloud side does not exist in NDI-python -- the coverage today isTestThroughARealDatabaseintests/test_cloud_series_reconstruction.py(real DID dispatch,fetch_cloud_filemocked). A live-cloud round-trip test would have caught this class of regression independently of the mocked path; also worth its own issue.What next
The paths forward for the underlying retirement item are documented in NDI-matlab#986's wrap-up (teach the handler to service local
sourcePaths, or change the location strategy). Not filing anything from here per the retraction scope; those decisions belong upstream.Generated by Claude Code