Trigger localization of clean room objects - #2826
Closed
francoisferrand wants to merge 2 commits into
Closed
Conversation
In a clean room, objects are created locally but their metadata still points at the source cluster's location: the data itself has not been copied over yet. Something has to notice those objects and ask for the data to be pulled in. The queue populator is the natural place for it, since bootstrap, re-bootstrap and streamed updates all go through the same oplog. When an object lands on a location flagged isCRR, publish a copyLocation action on the data mover topic and let the existing data mover + transition merge pipeline do the actual copy. This is unrelated to replicationInfo, which describes replication of a *local* object to remote sites, so the check sits before any replication condition. Activation is simply the presence of extensions.replication.localization.toLocation in the config; without it nothing changes. The populator only knows the object owner's canonical id, and resolving an account id per entry would throttle the whole populator, so the lookup is deferred to the transition processor, which now resolves it once up front and stamps it back on the action - the same way the garbage collector already does. Issue: BB-814
Codecov Report❌ Patch coverage is Additional details and impacted files
... and 2 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.6 #2826 +/- ##
===================================================
- Coverage 75.77% 75.37% -0.40%
===================================================
Files 200 200
Lines 13922 13999 +77
===================================================
+ Hits 10549 10552 +3
- Misses 3363 3437 +74
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
AccountIdCache.isKnown() is also true for cached misses, where get() returns undefined. The localization flow would then carry on with target.accountId unset and surface a confusing "failed to get backbeat client" InternalError instead of the NoSuchEntity that actually caused it, making a bad canonical id hard to diagnose in production. Treat a cached miss as the error it is, and only stamp the resolved account id onto the entry when there is one. Issue: BB-814
Contributor
Author
|
Superseded by #2829 (branch renamed to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In a clean room, objects are created locally but their metadata still points at the source cluster's location: the data itself has not been copied over yet. Something has to notice those objects and ask for the data to be pulled in.
The queue populator is the natural place for that, since bootstrap, re-bootstrap and streamed updates all go through the same oplog, so a single code path covers all three. When an object lands on a location flagged
isCRR, we publish acopyLocationaction on the data mover topic and let the existing data mover + transition merge pipeline do the actual copy.The check sits before any replication condition, on purpose:
replicationInfodescribes replication of a local object to remote sites, and the clean room copy resets it since the copy has not been replicated anywhere. A clean room site may also legitimately have its own forward replication rules. Localization is not tracked there either, updatinglocation/dataStoreNameis enough.Activation is simply the presence of
extensions.replication.localization.toLocationin the config (EXTENSIONS_REPLICATION_LOCALIZATION_TO_LOCATION). Without it the branch is inert and nothing changes, which is why it is deliberately absent fromconf/config.json.Account id resolution
The populator only knows the object owner's canonical id, and a Vault lookup per entry would throttle the whole populator. So the action carries
target.owneronly, and the lookup is deferred to the transition processor, which resolves it once up front and stamps it back onto the action, the same way the garbage collector already does. Lifecycle-originated actions always carry an account id, so the lookup never runs for them, and the GC entry emitted after the merge gets the resolved id rather than resolving it again.Metrics
Two new counters,
s3_backbeat_populator_localization_objects_totaland..._bytes_total, kept separate from the CRR site metrics so localization does not get conflated with forward replication.Skipped entries
Master keys (clean room buckets are versioned, the master is repaired by the metadata layer), delete markers, entries without a
dataStoreName(partial oplog projections), and objects with no location, since a 0-byte object has nothing to localize.isCRRwith a non-zero content length and no location is inconsistent metadata and gets logged as an error.Duplicates are expected and harmless: the topic is keyed per object, the data mover dedupes in memory,
CopyLocationTaskskips already-transitioned versions, and the merge task skips duplicate location updates.Not in this PR
transitionInProgress_garbageCollectLocationpublishes adeleteDatafor the old location, which against anisCRRsource would delete source data.Note that the full unit suite could not be run locally (needs Kafka/Redis); lint and the replication, gc, queuePopulator, lib and lifecycle task specs are green.
Issue: BB-814