Skip to content

RHINENG-27056: adding major migration runbook + doc updates#2244

Open
TenSt wants to merge 1 commit into
RedHatInsights:masterfrom
TenSt:stepan/RHINENG-27056-major-migration-runbook
Open

RHINENG-27056: adding major migration runbook + doc updates#2244
TenSt wants to merge 1 commit into
RedHatInsights:masterfrom
TenSt:stepan/RHINENG-27056-major-migration-runbook

Conversation

@TenSt

@TenSt TenSt commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

This PR:

  • Adds major database migration runbook for ops/engineers (deploy flow, flags, logs, diagnostics, rollback)
  • Documents db-migration Job + check-for-db init, session handling, and terminate_db_sessions flag
  • Links runbook from database.md, architecture.md, and AGENTS.md

Summary by Sourcery

Document major database migration operations and link a new runbook from existing docs for agents and operators.

Documentation:

  • Add a detailed major database migration runbook covering deploy flow, flags, logging, diagnostics, and rollback procedures.
  • Describe the db-migration Job and check-for-db init container behavior, including session handling and migration-related flags in database and architecture docs.
  • Link the new major migration runbook from AGENTS, database, and architecture documentation to guide agents and ops to the correct procedures.

@TenSt TenSt requested a review from a team as a code owner June 22, 2026 12:03
@sourcery-ai

sourcery-ai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds a detailed major database migration runbook and wires it into existing docs so ops/engineers understand the db-migration Job, init-container flow, session handling, and flags like terminate_db_sessions, plus where to find related code and diagnostics.

File-Level Changes

Change Details Files
Introduce a dedicated major database migration runbook documenting deploy flow, flags, logging, diagnostics, and rollback.
  • Create docs/md/major-migration-runbook.md with step-by-step guidance for heavy DDL migrations
  • Document DATABASE_ADMIN_CONFIG flags (schema_migration, force_migration_version, terminate_db_sessions, etc.) and when to use them
  • Describe db-migration Job behavior, retries, timeouts, and interaction with check-for-db init containers
  • Add Postgres diagnostic SQL snippets for advisory locks, app sessions, DDL locks, and migration state
docs/md/major-migration-runbook.md
Enhance database documentation to describe the migration mechanism and reference the new runbook and relevant configuration.
  • Add a Migrations section explaining where migration files live and how database-admin applies them in production
  • Document the db-migration ClowdApp Job and check-for-db init container behavior
  • Describe pre-DDL NOLOGIN behavior, optional terminate_db_sessions usage, and link to the major migration runbook for procedures and flags
  • Add references to ClowdApp parameters and config locations for migrations
docs/md/database.md
Update architecture docs to reference the db-migration Job, init-container behavior, and major migration runbook.
  • Clarify that in production only the db-migration Job runs migrations once per deploy
  • Document that other components wait in check-for-db init until the schema is current
  • Link to the database migrations section and major migration runbook from the database-admin description
docs/md/architecture.md
Expand AGENTS guide to point agents to the new migration runbook and summarize key operational behavior.
  • Augment docs links table to include database layout and migrations plus the major migration operations runbook
  • Add a section summarizing major DDL migrations for agents, covering deploy model, session handling, terminate_db_sessions usage, and logging cues
  • Emphasize advisory usage guidance for terminate_db_sessions and directing users to the runbook for detailed procedures
AGENTS.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@TenSt TenSt force-pushed the stepan/RHINENG-27056-major-migration-runbook branch from bb3ca4c to 0ea7852 Compare June 22, 2026 12:07
@codecov-commenter

codecov-commenter commented Jun 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.06%. Comparing base (b2d5a7d) to head (19be2c0).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2244   +/-   ##
=======================================
  Coverage   59.06%   59.06%           
=======================================
  Files         138      138           
  Lines        8848     8848           
=======================================
  Hits         5226     5226           
  Misses       3076     3076           
  Partials      546      546           
Flag Coverage Δ
unittests 59.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TenSt TenSt force-pushed the stepan/RHINENG-27056-major-migration-runbook branch from 0ea7852 to a3b04f6 Compare June 23, 2026 09:17
@TenSt TenSt force-pushed the stepan/RHINENG-27056-major-migration-runbook branch from a3b04f6 to 19be2c0 Compare June 23, 2026 09:24
@xbhouse xbhouse self-assigned this Jun 23, 2026

### DDL lock diagnostics

Replace `system_inventory` with the table your migration touches:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm i don't see a table name referenced in the query following this instruction. is the diagnostic query meant to filter on the table name?

AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
AND blocking_locks.pid != blocked_locks.pid
JOIN pg_stat_activity blocking ON blocking.pid = blocking_locks.pid
WHERE NOT blocking_locks.granted;

@xbhouse xbhouse Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compared this to the query AWS provided in slack and used cursor to help me understand 😅 it looks like the final WHERE clause might need to be blocked_locks.granted vs blocking_locks.granted, or it could be omitted since the JOIN already includes the NOT blocked_locks.granted condition

Comment thread docs/md/database.md
- A single **db-migration** ClowdApp Job runs `migrate` once per deploy (`completions: 1`, `parallelism: 1`).
- Manager, listener, evaluator, and other components use a **check-for-db** init container that polls until the schema matches (`database_admin/check-upgraded.sh`).

Before DDL, database-admin sets app users (`listener`, `evaluator`, `manager`, `vmaas_sync`) to `NOLOGIN`, waits for sessions to drain, optionally terminates lingering backends (`terminate_db_sessions`), runs `MigrateUp`, then restores `LOGIN`.

@xbhouse xbhouse Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is probably me stating the obvious but since the terminate_db_sessions flag is in another branch with additional docs there, this will just need to be rebased once that's merged :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants