ARSN-617: clean-read filter hiding non-localized versions - #2688
Draft
DarkIsDude wants to merge 6 commits into
Draft
ARSN-617: clean-read filter hiding non-localized versions#2688DarkIsDude wants to merge 6 commits into
DarkIsDude wants to merge 6 commits into
Conversation
Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a per-call `cleanRead` flag on the metadata read and listing APIs, hiding the object versions whose data still lives on a remote source site (`locations[objMD.dataStoreName].isCRR`), as needed by the clean-room D/R deployments. MongoClientInterface implements it in the MongoDB query itself, so that listing limits and truncation stay correct, and MetadataWrapper sets the flag on every read and listing call when the deployment enables it. Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Hello darkisdude,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## development/8.5 #2688 +/- ##
===================================================
+ Coverage 74.66% 74.76% +0.09%
===================================================
Files 227 227
Lines 18606 18650 +44
Branches 3880 3862 -18
===================================================
+ Hits 13892 13943 +51
+ Misses 4709 4702 -7
Partials 5 5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Rename the per-call flag to `hideNonLocalizedVersions`, keep the callback as the last argument of `getLatestVersion`, and trim the comments down to the non-obvious ones. Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A master pointing at a non-localized version is no longer hidden: the master key is resolved on the way out of the listing stream, exposing the newest localized version of the object — the same resolution a PHD key goes through, and the same one getObject already performs, so a listing and a GET agree. Only an object with no localized version at all is dropped. Issue: ARSN-617 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Part of the clean-room D/R feature (epic ZENKO-5238, design Artesca Full D/R).
In a clean room, metadata is replicated before the object data is copied locally: until then the version's
dataStoreNamestill refers to the remote production site. Such a version is non-localized (locations[objMD.dataStoreName].isCRR) and must not be observable by the clients.What this adds
A per-call
hideNonLocalizedVersionsflag on the metadata read and listing APIs (getObject,getObjects,getBucketAndObject,listObject,listMultipartUploads) — a backend-agnostic contract, implemented byMongoClientInterfaceonly for now.MetadataWrappersets it on every read and listing call when the deployment enables it, and refuses to start if the backend does not implement it. The location configuration is injected through a newgetLocationConstraintsparameter, evaluated per call so configuration updates are picked up.Behavior when the flag is set:
GET/HEAD/GetObjectAttributesby versionId and batch gets exclude the non-localized versions;getObjectresolves it identically, so a listing and aGETagree. An object with no localized version at all is dropped;dataStoreNameand are never hidden.Version-level filtering happens in the MongoDB query (
$nin), added as an$andelement after the metadata-search query is merged so a search ondataStoreNamecan neither overwrite it nor be overwritten by it. The master stream is not filtered, since the masters must reach the resolution step.Backbeat-internal listings (
listLifecycleObject) and every write path are untouched. Activation lives elsewhere: CLDSRV-957 (Cloudserver configuration) and ZKOP-564 (user-facing Cloudserver only).Interaction with ARSN-618
ARSN-618 answers the same "which version is current" question at write time (never move the master onto a non-localized version), unconditionally. It can reuse the condition helpers and the
getLatestVersionfilter parameter added here, and it must update thegetLatestVersion(..., null, ...)call sites on the write and delete paths so a repair promotes the newest localized version.The read-time resolution added here does not become redundant: it also covers the entries that never traverse the write path, namely the mongodump/mongorestore bootstrap and re-bootstrap loads. Note that the design rejected read-time master filtering as an alternative, so the design and ARSN-618 should record that read-time resolution now exists.
Two items for ARSN-618:
filter_onNewMasterKeyV0would then treat a version key as a new master and emitkey\0<versionId>as a listing entry. The guard added here prevents exactly that, but only when the flag is set, while ARSN-618 is unconditional — so the internal Cloudserver would be exposed. ARSN-618 should make the guard unconditional or keep a PHD-style master. v1 is unaffected.hideNonLocalizedVersions.spec.jscreates a non-localized master throughputObjectMD. Once ARSN-618 lands, that write no longer moves the master, so the fixtures stop producing the state under test and must insert the documents directly.Tests
getLatestVersion) and the$andmerge against a colliding search query;MetadataWrapperflag injection, its absence on internal and write calls, and the unsupported-backend refusal;mongodb-memory-server, v0 and v1): mix of localized versions, non-localized versions, a delete marker and an object whose current version is not localized — master and version listings, both paged withmaxKeys: 1,getObjectby version and by master, batch gets, and every case re-run with the flag off.The prettier commits are format-only, kept separate from the functional changes.
🤖 Generated with Claude Code