Download object store directory keys such as zarr stores

[zaihuaji]

Summary

• object_copy_local() now recognises a key that is a prefix over many objects (a zarr store) and hands it to the new object_copy_local_directory(), which lists the prefix and downloads every member into its place under the target directory.
• The single object download is factored out into object_get_local(), taking the expected size as an argument so the member sizes returned by object_glob() are reused.
• Version bumped to 3.0.16.

Why

dsquasar -A 3 failed on d694517 with isd_s3_cli go -k .../....zarr -b gdex-data returning a 404 on HeadObject: the key is a prefix, not an object. The upload side already handled this in check_object_file(); the download side did not.

Note

Rechecking each member key on its own would misread chunk names: listing is a prefix match, so time/1 also returns time/10 and time/11, which check_object_file() would classify as a directory. The sizes from object_glob() are used instead, which also halves the round trips.

Test plan

• py_compile clean
• Stubbed object_glob/object_get_local run: nested member paths preserved, sibling prefix x.zarr2/... excluded, trailing / on the prefix stripped
• Rerun dsquasar -A 3 on d694517 against Boreas

[Copilot]

🟡 Changes recommended

Unresolved download-path issues can cause failures or leave the legacy API without the fix.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds support for downloading object-store prefixes, such as Zarr stores, as local directory trees.

Changes:

• Downloads nested prefix members recursively.
• Reuses listed object sizes for validation.
• Updates the package version to 3.0.16.

File summaries

File	Summary
src/rda_python_common/pg_file.py	Adds prefix downloads, but legacy API handling, directory markers, and exact-key classification still require fixes.
src/rda_python_common/__init__.py	Updates the package version.
README.md	Updates the documented version.
pyproject.toml	Updates the project version.

Review details

Suppressed comments (5)

src/rda_python_common/pg_file.py:870

• The relative key suffix is concatenated into a local path without validation. A valid object key such as prefix/../../outside produces todir/../../outside, so the download escapes todir and can overwrite files outside the requested destination. Reject absolute paths and .. components, or verify the normalized destination remains under todir, before downloading.

         tofile = "{}/{}".format(todir, key[plen:])
         if not self.object_get_local(tofile, key, flist[key]['data_size'], bucket, logact): return self.FAILURE

src/rda_python_common/pg_file.py:870

• These keys come from object_glob() and are passed into object_get_local(), which interpolates them into the isd_s3_cli go -k ... command. A store member containing shell metacharacters (for example ;) can therefore turn a listed object name into command injection when pgsystem() uses a shell. Pass argv arguments to the subprocess or shell-quote every command argument before downloading listed keys.

         if not self.object_get_local(tofile, key, flist[key]['data_size'], bucket, logact): return self.FAILURE

src/rda_python_common/pg_file.py:825

• The downloaded file is fromname in the current target directory, but this call passes the remote object key fromfile to set_local_mode(). When the desired mode differs, it attempts to chmod a path such as prefix/nested/member relative to the target directory, so the downloaded member keeps the wrong permissions (and the ignored return value hides the failure). Use the local fromname here.

            if info['data_size'] == fsize:
               self.set_local_mode(fromfile, info['isfile'], 0, info['mode'], info['logname'], logact)

src/rda_python_common/pg_file.py:864

• An object-store directory marker such as fromdir + '/' (or any nested key ending in /) also satisfies this filter. Its suffix is empty/trailing, so the next line builds todir/ and object_get_local() uses an empty basename; check_local_file('') then fails and aborts the whole directory download. Exclude marker keys or handle them as directories.

      keys = [key for key in flist if key.startswith(prefix)]

src/rda_python_common/pg_file.py:789

• This routes every check_object_file() result with isfile == 0 into the directory path, but check_object_file() also marks an exact object as non-file when prefix listing returns similarly named keys (for example time/1 alongside time/10). object_copy_local_directory() then filters for time/1/, finds no members, and reports the valid exact object as missing. Make the object-vs-prefix classification delimiter-aware or handle an exact-key match before treating the result as a directory.

      if not finfo['isfile']: return self.object_copy_local_directory(tofile, fromfile, bucket, logact)
      return self.object_get_local(tofile, fromfile, finfo['data_size'], bucket, logact)

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.