Skip to content

ARSN-617: clean-read filter hiding non-localized versions - #2688

Draft
DarkIsDude wants to merge 6 commits into
development/8.5from
improvement/ARSN-617/clean-read-filter
Draft

ARSN-617: clean-read filter hiding non-localized versions#2688
DarkIsDude wants to merge 6 commits into
development/8.5from
improvement/ARSN-617/clean-read-filter

Conversation

@DarkIsDude

@DarkIsDude DarkIsDude commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Part of the clean-room D/R feature (epic ZENKO-5238, design Artesca Full D/R).

In a clean room, metadata is replicated before the object data is copied locally: until then the version's dataStoreName still refers to the remote production site. Such a version is non-localized (locations[objMD.dataStoreName].isCRR) and must not be observable by the clients.

What this adds

A per-call hideNonLocalizedVersions flag on the metadata read and listing APIs (getObject, getObjects, getBucketAndObject, listObject, listMultipartUploads) — a backend-agnostic contract, implemented by MongoClientInterface only for now. MetadataWrapper sets it on every read and listing call when the deployment enables it, and refuses to start if the backend does not implement it. The location configuration is injected through a new getLocationConstraints parameter, evaluated per call so configuration updates are picked up.

Behavior when the flag is set:

  • version listings, GET/HEAD/GetObjectAttributes by versionId and batch gets exclude the non-localized versions;
  • a master listing keeps the object and shows its newest localized version — the master key is resolved on the way out of the stream, the same way a PHD key already is, and getObject resolves it identically, so a listing and a GET agree. An object with no localized version at all is dropped;
  • delete markers and PHD keys carry no dataStoreName and are never hidden.

Version-level filtering happens in the MongoDB query ($nin), added as an $and element after the metadata-search query is merged so a search on dataStoreName can neither overwrite it nor be overwritten by it. The master stream is not filtered, since the masters must reach the resolution step.

Backbeat-internal listings (listLifecycleObject) and every write path are untouched. Activation lives elsewhere: CLDSRV-957 (Cloudserver configuration) and ZKOP-564 (user-facing Cloudserver only).

Interaction with ARSN-618

ARSN-618 answers the same "which version is current" question at write time (never move the master onto a non-localized version), unconditionally. It can reuse the condition helpers and the getLatestVersion filter parameter added here, and it must update the getLatestVersion(..., null, ...) call sites on the write and delete paths so a repair promotes the newest localized version.

The read-time resolution added here does not become redundant: it also covers the entries that never traverse the write path, namely the mongodump/mongorestore bootstrap and re-bootstrap loads. Note that the design rejected read-time master filtering as an alternative, so the design and ARSN-618 should record that read-time resolution now exists.

Two items for ARSN-618:

  • v0 listings with the flag off. ARSN-618 leaves the master absent when no localized version exists. In v0 the master and version keys share one key range, so filter_onNewMasterKeyV0 would then treat a version key as a new master and emit key\0<versionId> as a listing entry. The guard added here prevents exactly that, but only when the flag is set, while ARSN-618 is unconditional — so the internal Cloudserver would be exposed. ARSN-618 should make the guard unconditional or keep a PHD-style master. v1 is unaffected.
  • Test fixtures. hideNonLocalizedVersions.spec.js creates a non-localized master through putObjectMD. Once ARSN-618 lands, that write no longer moves the master, so the fixtures stop producing the state under test and must insert the documents directly.

Tests

  • unit: generated queries (flag on/off, no CRR location, no location configuration, filter propagation to getLatestVersion) and the $and merge against a colliding search query; MetadataWrapper flag injection, its absence on internal and write calls, and the unsupported-backend refusal;
  • functional (mongodb-memory-server, v0 and v1): mix of localized versions, non-localized versions, a delete marker and an object whose current version is not localized — master and version listings, both paged with maxKeys: 1, getObject by version and by master, batch gets, and every case re-run with the flag off.

The prettier commits are format-only, kept separate from the functional changes.

🤖 Generated with Claude Code

DarkIsDude and others added 2 commits August 19, 2026 15:54
Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a per-call `cleanRead` flag on the metadata read and listing APIs,
hiding the object versions whose data still lives on a remote source
site (`locations[objMD.dataStoreName].isCRR`), as needed by the
clean-room D/R deployments. MongoClientInterface implements it in the
MongoDB query itself, so that listing limits and truncation stay
correct, and MetadataWrapper sets the flag on every read and listing
call when the deployment enables it.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bert-e

bert-e commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Hello darkisdude,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue ARSN-617 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 8.5.15

Please check the Fix Version/s of ARSN-617, or the target
branch of this pull request.

@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.13158% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.76%. Comparing base (d6ef4ed) to head (ae474fa).
⚠️ Report is 3 commits behind head on development/8.5.

Files with missing lines Patch % Lines
lib/storage/metadata/mongoclient/readStream.js 73.07% 7 Missing ⚠️
lib/algos/list/delimiterMaster.ts 90.74% 5 Missing ⚠️
...orage/metadata/mongoclient/MongoClientInterface.ts 94.11% 3 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                 @@
##           development/8.5    #2688      +/-   ##
===================================================
+ Coverage            74.66%   74.76%   +0.09%     
===================================================
  Files                  227      227              
  Lines                18606    18650      +44     
  Branches              3880     3862      -18     
===================================================
+ Hits                 13892    13943      +51     
+ Misses                4709     4702       -7     
  Partials                 5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DarkIsDude DarkIsDude self-assigned this Aug 24, 2026
@bert-e

bert-e commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue ARSN-617 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 8.5.16

Please check the Fix Version/s of ARSN-617, or the target
branch of this pull request.

DarkIsDude and others added 3 commits August 27, 2026 16:21
Rename the per-call flag to `hideNonLocalizedVersions`, keep the
callback as the last argument of `getLatestVersion`, and trim the
comments down to the non-obvious ones.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A master pointing at a non-localized version is no longer hidden: the
master key is resolved on the way out of the listing stream, exposing
the newest localized version of the object — the same resolution a PHD
key goes through, and the same one getObject already performs, so a
listing and a GET agree. Only an object with no localized version at
all is dropped.

Issue: ARSN-617

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bert-e

bert-e commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue ARSN-617 contains:

  • 8.5.15

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 8.5.16

Please check the Fix Version/s of ARSN-617, or the target
branch of this pull request.

@bert-e

bert-e commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants