initial commit for multidb - #7862
Open
YasenT wants to merge 4 commits into
Open
Conversation
Contributor
Author
|
/retest |
Contributor
Author
|
@gerrod3 What do you think? Can I get a review? |
YasenT
marked this pull request as ready for review
July 24, 2026 10:30
gerrod3
reviewed
Aug 3, 2026
gerrod3
left a comment
gerrod3
left a comment
Contributor
There was a problem hiding this comment.
Round 1 of reviews. This is honestly quite unreviewable in its current state. The AI comments are a nightmare and make references to docs and comments I don't have access to. The commits are not logically structured either. I would probably have ordered them something like:
- The initial adding of the database-alias and db-router
- Fixing management commands and other models (GenericReleation)
- Adding the database domain migration command
- CI work and tests
I would like to set expectations now that this will require major changes and many iterations before we are close to a state that might be mergeable.
Comment on lines
+318
to
+323
| CROSS_PLANE_RECONCILIATION_GRACE_MINUTES = 60 | ||
|
|
||
| # KI-11: how long, in days, a confirmed-orphaned cross-plane row is kept (logged/alerted on every | ||
| # sweep) before the reconciliation sweep deletes it outright. 0 disables purging entirely -- | ||
| # orphans are only ever logged, never deleted, which is the safe default. | ||
| CROSS_PLANE_RECONCILIATION_PURGE_AFTER_DAYS = 0 |
Contributor
There was a problem hiding this comment.
Why do we need these two different settings?
Contributor
Author
There was a problem hiding this comment.
The first one makes sure we don't "flag" objects in "flight" as orphans. For example in the case of a migration of an active domain to a new DB, we need some grace period before we start flagging as orphans
And than how regularly we purge is a different setting, the 2x can be very different values. Multiple days vs multiple hours. Makes sense to be 2 phased in a way.
| _DOMAIN_WALK_MAX_DEPTH = 2 | ||
|
|
||
|
|
||
| def _resolve_domain_id(value, _depth=0, _seen=None): |
Contributor
There was a problem hiding this comment.
Do we really need this? Yeah it's probably the safest way to get the domain of the object, but I would expect that get_domain would always return the correct domain that the object is in. Maybe it doesn't matter since creating GenericRelationships typically never happen in a hot path
Contributor
Author
There was a problem hiding this comment.
I see it more as a protection for the future, just playing safe.
Contributor
|
Rebase this please |
YasenT
force-pushed
the
multidb-implementation
branch
from
August 10, 2026 10:20
ed36dcc to
2ad60ec
Compare
Contributor
Author
rebased. tests re-running, had to fix few things, seems okay for now |
YasenT
force-pushed
the
multidb-implementation
branch
from
August 13, 2026 19:08
35804e0 to
6c71cca
Compare
YasenT
force-pushed
the
multidb-implementation
branch
from
September 1, 2026 17:21
6c71cca to
dd289cb
Compare
…ting Introduce the core routing infrastructure needed to split a Pulp instance's domains across multiple database aliases: a `database_alias` (and `moving`) field on `Domain`, a `PulpDomainRouter` that pins control-plane models to `default` and routes data-plane models to their owning domain's alias (via instance hints or the domain ContextVar), and a `CrossDBQuerySetMixin` to safely materialize subqueries that would otherwise span two databases. Also add the supporting ContextVar/util helpers (`with_migration_alias`, `domain_db`, `for_each_domain`, and a `get_domain_pk()` alias fix) used by later commits, and the app-startup guards needed for `migrate` and its existing post_migrate hooks to behave correctly once more than one database alias is configured. Co-authored-by: Cursor <cursoragent@cursor.com>
…ting Update existing management commands (datarepair, datarepair-2327, remove-plugin, repository-size, rotate-db-key, dump-publications-to-fs, handle-artifact-checksums, analyze-publication) to iterate per-domain and query the correct database alias instead of assuming a single database. Fix GenericForeignKey/GenericRelation handling across database boundaries: resolve the transitive domain for content_object targets that don't carry their own pulp_domain field (e.g. RepositoryVersion), add a content_object_domain column to CreatedResource/ExportedResource/ UserRole/GroupRole to record which alias the referenced object actually lives on, and clean up UserRole/GroupRole rows when their cross-plane target is deleted (Django's native GenericRelation cascade can't follow a relation across two different databases). Also fix ProfileArtifact's cross-plane artifact lookup in the task profiling viewset, and materialize a couple of querysets in role_util.py that would otherwise become invalid cross-database subqueries. Co-authored-by: Cursor <cursoragent@cursor.com>
Add pulpcore-manager migrate-all/migrate for orchestrating Django migrations across every configured database alias in the right order. Add move-domain/cleanup-moved-domain for relocating a domain's data-plane rows from one alias to another: begin with a read-only cutover, copy and verify the data, then complete the cutover and delete the stale copy left behind on the source alias. Add sync-domains/domain-size for keeping Domain metadata replicated to satellite aliases and reporting on a domain's footprint, and reconcile-cross-plane-references for sweeping up orphaned cross-plane references left behind by these operations. This is supported by MigrationStatus/DomainMove bookkeeping models, Domain row replication to satellites (domain_sync.py), advisory locks so only one orchestrator/mover runs at a time, per-alias connectivity and migration-completeness reporting on the /status/ endpoint, worker gating so tasks are deferred while a domain's data is mid-move or its alias is unreachable, and a degraded API response for the same cases. Co-authored-by: Cursor <cursoragent@cursor.com>
Add a "multi_db" CI matrix leg that spins up a second Postgres service container and runs the unit test suite against it with PulpDomainRouter registered. Add unit tests covering the router itself, Domain replication/reconciliation, domain move/cleanup, cross-plane reference reconciliation, and the DomainMiddleware/task changes from the preceding commits. Also fix flakiness in test_cancel_task_group by retrying task-group cancellation on a transient 409 instead of failing outright. Co-authored-by: Cursor <cursoragent@cursor.com>
YasenT
force-pushed
the
multidb-implementation
branch
from
September 1, 2026 17:27
dd289cb to
124b368
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial PR for the multidb implementation. To test github actions. etc