Skip to content

Retire the manifest cloud-reference workaround (companion to VH-Lab/NDI-matlab#986) #300

Closed as not planned

Description

@stevevanhooser

Companion to VH-Lab/NDI-matlab#986. Ports the workaround retirement now that VH-Lab/DID-python#88 has landed the reads that make it unnecessary.

What the workaround is

ndi.cloud.filehandler.updateFileInfoForLocalFiles rewrites a downloaded document's file_info to point at the local files that came down with it. For a series MANIFEST it did two things a plain file does not:

  1. Set the local file location (same as any file).
  2. Append a second ndicloud location ndic://<cloudDatasetId>/<manifestUid> alongside.

The second location was there to give DID's custom_file_handler a non-file source path for member fetches. Without it, download_file_from_cloud was handed a plain local path and returned without producing anything, so every member of a downloaded series read as absent -- silently, because DID reads a handler that fails as a plain miss. See VH-Lab/NDI-matlab#966.

Why it can go now

That makes the second location dead weight -- DID never consults it anymore.

What to change

src/ndi/cloud/filehandler.py:

  • In updateFileInfoForLocalFiles, drop the block that appends _ndic_location_record(cloud_dataset_id, uid) to a manifest's locations. Simplify the write-back around it (no more grown-list case).
  • Remove the now-unused helpers _ndic_location_record and _declared_series_names.
  • Update the docstring: cloud_dataset_id still drives the ingest_locations rebuild via reconstructSeriesIngestLocations (Cloud-downloaded documents refused by DID's MembersNotLocatable guard VH-Lab/NDI-matlab#958), but no longer the manifest's second location.
  • Comment update at the call site in src/ndi/cloud/orchestration.py.

tests/test_cloud_series_reconstruction.py:

  • Remove TestTheManifestKeepsItsCloudReference (analog of TestDownloadedSeriesManifestKeepsItsCloudLocation.m; its whole subject was pinning the second location).
  • Remove test_a_member_of_a_downloaded_series_can_be_read -- it asserted the handler receives an ndic:// URI built from the manifest's second location, which no longer exists. End-to-end coverage of a member of a downloaded series lives on the cloud side, mirroring NDI-matlab's testMembersSurviveADownloadFromTheCloud.
  • Add TestSeriesManifestHasExactlyOneLocation as the regression guard: exactly one location, of type file. This mirrors TestUpdateFileInfoPreservesSeries.testSeriesManifestHasExactlyOneLocation added in Retire the manifest cloud-reference workaround (#986) VH-Lab/NDI-matlab#989.

Bridge yamls:

  • Bump matlab_last_sync_hash on updateFileInfoForLocalFiles to the NDI-matlab#989 head (2ad46c0f) and drop the "second location beside the local copy" language from the decision_log.
  • Companion notes in src/ndi/cloud/sync/ndi_matlab_python_bridge.yaml.

Related work, deliberately out of scope

VH-Lab/NDI-matlab#988 also taught the SyncFiles=false path (updateFileInfoForRemoteFiles) to rebuild series ingest_locations by fetching the manifest bytes through the DID customFileHandler contract. NDI-python's updateFileInfoForRemoteFiles does not yet do this -- a document with a populated series returned from a SyncFiles=false cloud round trip will hit DID's MembersNotLocatable guard on the following add_docs. Separate issue, separate change.

References

Activity

  1. stevevanhooser commented on Sep 14, 2026

    @stevevanhooser
    ContributorAuthor

    Closing as not planned. The MATLAB-side retirement was retracted (VH-Lab/NDI-matlab#989 closed unmerged, VH-Lab/NDI-matlab#986 closed as not planned), and the same premise fails on the Python side.

    Why

    The premise for retiring the workaround was that DID (post DID-matlab#191 / DID-python#88) could reach a manifest's local file location on a member fetch and hand it to the customFileHandler. It can -- but "offering" a sourcePath is not "servicing" one. DID iterates every manifest location and offers the local-file one to the handler; the NDI handler (in both languages) refuses any sourcePath that does not start with ndic://. Without the second ndicloud location on the manifest, the handler has nothing to service and every member of a downloaded series (SyncFiles=true) reads as absent -- silently, because DID reads a handler that produces nothing as a plain miss.

    Cloud CI on NDI-matlab#989 caught this on FileSeriesRoundTripTest.testMembersSurviveADownloadFromTheCloud (SyncFiles=true) and .testTheBatchScopeIsTheSeriesAndNotTheWholeDocument, both failing with DID:SQLITEDB:open, chunkdata.bin_1 ... cannot be accessed.

    The NDI-python analog ndi.cloud.filehandler.download_file_from_cloud (src/ndi/cloud/filehandler.py:677-678) has the same behavior:

    if not str(source_path).startswith(NDIC_SCHEME):
        return

    so the regression would have hit here too.

    Full analysis on the NDI-matlab side:

    State of the world

    • Companion PR Retire the manifest cloud-reference workaround (#300) #301: closed without merging. origin/main still carries matlab_last_sync_hash: "504c5bcf4" for updateFileInfoForLocalFiles in both bridge YAMLs, so nothing needs reverting.
    • Workaround stays: updateFileInfoForLocalFiles still appends the second ndicloud location on a series manifest, _ndic_location_record and _declared_series_names stay, and tests/test_cloud_series_reconstruction.py::TestThroughARealDatabase::test_a_member_of_a_downloaded_series_can_be_read remains as the guard against removing the workaround silently.

    Coverage note (out-of-scope item from this issue)

    This issue's "Related work, deliberately out of scope" section already recorded that NDI-python's updateFileInfoForRemoteFiles does not rebuild series ingest_locations on the SyncFiles=false path via the DID customFileHandler contract (NDI-matlab#988's addition). Retracting this retirement does not change that gap, and there is no NDI-python test analogous to a SyncFiles=false / cloud-only-manifest read path (offline + cloud) matching NDI-matlab#987/#988. That gap is worth its own tighter issue when it is picked up, and is not being folded back into this closed issue.

    Similarly, an integration test analogous to NDI-matlab's FileSeriesRoundTripTest.testMembersSurviveADownloadFromTheCloud on the live cloud side does not exist in NDI-python -- the coverage today is TestThroughARealDatabase in tests/test_cloud_series_reconstruction.py (real DID dispatch, fetch_cloud_file mocked). A live-cloud round-trip test would have caught this class of regression independently of the mocked path; also worth its own issue.

    What next

    The paths forward for the underlying retirement item are documented in NDI-matlab#986's wrap-up (teach the handler to service local sourcePaths, or change the location strategy). Not filing anything from here per the retraction scope; those decisions belong upstream.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions