Skip non-localized entries in replication and lifecycle - #2832
Draft
francoisferrand wants to merge 5 commits into
Draft
Skip non-localized entries in replication and lifecycle#2832francoisferrand wants to merge 5 commits into
francoisferrand wants to merge 5 commits into
Conversation
In a clean room, objects are created locally but their metadata still points at the source cluster's location: the data itself has not been copied over yet. Something has to notice those objects and ask for the data to be pulled in. The queue populator is the natural place for it, since bootstrap, re-bootstrap and streamed updates all go through the same oplog. When an object lands on a location flagged isCRR, publish a copyLocation action on the data mover topic and let the existing data mover + transition merge pipeline do the actual copy. This is unrelated to replicationInfo, which describes replication of a *local* object to remote sites, so the check sits before any replication condition. Activation is simply the presence of extensions.replication.localization.toLocation in the config; without it nothing changes. The populator only knows the object owner's canonical id, and resolving an account id per entry would throttle the whole populator, so the lookup is deferred to the transition processor, which now resolves it once up front and stamps it back on the action - the same way the garbage collector already does. Issue: BB-814
AccountIdCache.isKnown() is also true for cached misses, where get() returns undefined. The localization flow would then carry on with target.accountId unset and surface a confusing "failed to get backbeat client" InternalError instead of the NoSuchEntity that actually caused it, making a bad canonical id hard to diagnose in production. Treat a cached miss as the error it is, and only stamp the resolved account id onto the entry when there is one. Issue: BB-814
Initialising the vault client wrapper for every processor using assume role auth broke the expiration processor, which has no vaultAdmin in its config: start() threw on the missing host, and isReady() would have waited forever on credentials nobody was fetching. Only the transition processor receives actions without an account id, so gate both on a vault endpoint actually being configured, and fail the lookup explicitly if one is ever requested without it. Also add the new clean room location to the circuit breaker probe expectations, which enumerate every configured location. Issue: BB-814
In a clean room, an object's metadata is local but its data still lives on the source cluster: dataStoreName points at an isCRR location. Such a version has nothing local to offer, so the existing workflows must leave it alone until localization has rewritten the metadata. The replication populator already diverted those entries to localization, but only when localization was configured; otherwise they fell through and got replicated, which would copy a location the target cannot read. Skip them in both cases. Nothing is lost: the localization merge produces a new oplog entry with the real location, and replication picks it up then. Lifecycle transitions get the same treatment. Localization is the only valid transition out of an isCRR location and it has its own trigger, so a transition rule matching such a version is simply skipped and the version re-evaluated on a later scan. Issue: BB-816
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 1 file with indirect coverage changes
@@ Coverage Diff @@
## improvement/BB-814 #2832 +/- ##
======================================================
- Coverage 75.64% 75.64% -0.01%
======================================================
Files 200 200
Lines 14004 14008 +4
======================================================
+ Hits 10594 10597 +3
- Misses 3400 3401 +1
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
francoisferrand
marked this pull request as ready for review
August 24, 2026 12:40
francoisferrand
force-pushed
the
improvement/BB-814
branch
from
August 25, 2026 20:58
8cd19f1 to
aedd849
Compare
francoisferrand
marked this pull request as draft
August 26, 2026 07:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2826 (BB-814), so the base here is
improvement/BB-814. This replaces #2827, which GitHub auto-closed when the branch was renamed to drop thefrancoisferrand-prefix.In a clean room, an object's metadata is local but its data still lives on the source cluster:
dataStoreNamepoints at anisCRRlocation. Such a version has nothing local to offer, so the existing workflows must leave it alone until localization has rewritten the metadata.BB-814 already diverted those entries in the replication populator to localization, but only when localization was configured. Otherwise they fell through and got replicated, which would copy a location the target cannot read. They are now skipped in both cases. Nothing is lost: the localization merge produces a new oplog entry with the real location, and replication picks it up then.
Lifecycle transitions get the same treatment. Localization is the only valid transition out of an
isCRRlocation and it is driven by its own path, not by lifecycle rules, so a transition rule matching such a version is skipped and the version re-evaluated on a later scan.LifecycleTaskV2only calls_applyTransitionRule, it does not override it, so the single check covers both tasks.Tested with the replication and lifecycle unit suites; the populator specs cover the disabled-localization and
PENDINGcases, and the new_applyTransitionRulespecs assert no data-mover action and notransitionInProgressfor anisCRRobject, with a regular location as positive control.Issue: BB-816