Core: Make psycopg (v3) the default synchronous Postgres driver

[Dev-iL]

related:

• Remove psycopg2 and migrate the sync Postgres driver to psycopg3 #68453
• Switch the default async Postgres driver from asyncpg to psycopg3 (core) #68496

Mirrors the async default switch: a bare postgresql:// or legacy postgres:// / postgres+psycopg2:// sql_alchemy_conn is now rewritten to postgresql+psycopg:// instead of postgresql+psycopg2://. An explicit postgresql+psycopg2:// URL is never rewritten and keeps working as long as apache-airflow-providers-postgres[psycopg2] is installed.

This PR touches core only - there will be separate PRs for providers.

What changed

• _upgrade_postgres_metastore_conn (airflow-core/src/airflow/configuration.py): good_scheme changed from postgresql+psycopg2 to postgresql+psycopg when the psycopg (v3) package is actually importable (find_spec("psycopg") is not None), mirroring the exact guard settings.py already uses for the async driver default (_USE_PSYCOPG3); falls back to postgresql+psycopg2 otherwise, so environments that don't have psycopg3 installed (e.g. a min-supported-Airflow-version test matrix, or a provider release pinned before psycopg3 support existed) keep working instead of hitting sqlalchemy.exc.NoSuchModuleError. bad_schemes is unchanged.
• prepare_engine_args (airflow-core/src/airflow/settings.py): the psycopg2-specific executemany_mode/executemany_batch_page_size tuning now follows the actually-configured SQL_ALCHEMY_CONN scheme, not whether the psycopg package happens to be importable.
• Two pre-existing migration bugs, invisible under psycopg2 but broken under psycopg3, found by running this branch's CI and then replaying the full upgrade/downgrade/upgrade chain against a real Postgres locally:
  • airflow-core/src/airflow/migrations/versions/0017_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py: literal("0") (a Python str) bound to the Integer column dag.max_consecutive_failed_dag_runs. psycopg2 let Postgres coerce the untyped bind param silently; psycopg3's dialect renders it with an explicit ::VARCHAR cast, causing psycopg.errors.DatatypeMismatch on every fresh migration. Changed to literal(0).
  • airflow-core/src/airflow/migrations/versions/0094_3_2_0_replace_deadline_inline_callback_with_fkey.py: the :prefix / :classname string bind params were passed directly into json_build_object(...), a polymorphic "any"-argument function. psycopg3 uses the extended query protocol, so Postgres must resolve each parameter's type at Parse time and can't for polymorphic functions without an explicit cast, raising psycopg.errors.IndeterminateDatatype. Fixed by wrapping both bind sites (upgrade and downgrade) in CAST(... AS text).
• scripts/ci/docker-compose/backend-postgres.yml: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN reverted from a hardcoded postgresql+psycopg://... back to the bare postgresql://... it was before this PR's docs/config sweep. This file is shared by every Postgres-backed CI job, including the "Migration Tests" job that runs against an old released Airflow (--use-airflow-version <MIN_AIRFLOW_VERSION>, currently 2.11.0) with --mount-sources skip — i.e. it never loads this branch's configuration.py at all. Hardcoding the new +psycopg scheme meant that job handed create_engine an explicit driver it doesn't have installed, causing sqlalchemy.exc.NoSuchModuleError: postgresql.psycopg, which the find_spec guard above can't fix since it doesn't run in that job. Reverting to bare postgresql:// lets each Airflow version's own scheme-upgrade logic pick a driver it actually has (+psycopg on this branch, +psycopg2 on 2.11.0). AIRFLOW__CELERY__RESULT_BACKEND is intentionally left as explicit +psycopg, since that line only affects the celery-specific integration job, which always runs current-branch code where psycopg3 is guaranteed present.
• Two more psycopg2-vs-psycopg3 behavioral differences found by re-running CI after the fixes above got the migration chain running again:
  • airflow-core/src/airflow/models/serialized_dag.py::get_dag_dependencies: the Postgres #> path-extraction operator (jsonb #> text[]) was fed an untyped Python string bind param shaped like a Postgres array literal ('{"dag","dag_dependencies"}'). psycopg2 let Postgres infer/cast it to text[] implicitly; psycopg3 renders it as character varying, which doesn't match the operator's signature, raising psycopg.errors.UndefinedFunction. Fixed by passing an explicit literal(["dag", "dag_dependencies"], type_=ARRAY(String)) instead of a string literal, so the array type is unambiguous regardless of driver — this broke the ui/dependencies API route and every test depending on it (test_serialized_dag.py, test_dependencies.py, test_structure.py, test_dag_command.py's show-dependencies CLI tests).
  • airflow-core/tests/unit/api_fastapi/common/test_exceptions.py: two tests asserted on the exact rendered SQL statement and driver error text embedded in a IntegrityError → HTTPException conversion. psycopg3 adds explicit ::VARCHAR/::INTEGER casts to bind params in the rendered statement (psycopg2 doesn't) and drops the trailing \n psycopg2 leaves on Postgres's DETAIL: error line. Since the sibling multi-column test already established the right pattern for this ("the SQL statement is an implementation detail, so we match on the statement pattern instead of an exact match"), applied the same treatment to the single-column test (pop statement, match on target table) and added .rstrip("\n") normalization for orig_error to both tests, since that trailing whitespace is equally a driver-rendering detail, not meaningful content.
  • airflow-core/tests/unit/core/test_configuration.py::test_upgrade_postgres_metastore_conn: this test asserted the real, ambient find_spec("psycopg") result rather than mocking it, so it silently depended on whatever the CI job's actual environment happened to have installed — it passed on the standard test image but failed on the "Low dep tests: core" job (which doesn't install psycopg3 for airflow-core's own test env). Added the same find_spec mock the sibling _without_psycopg3 test already uses, making the psycopg3-available assertion deterministic instead of environment-dependent.
• New sync-driver section in airflow-core/docs/howto/set-up-database.rst, mirroring the existing async section, documenting the new default and how to restore psycopg2 via the provider's [psycopg2] extra.
• Updated remaining hardcoded postgresql+psycopg2:// example connection strings across docs and reference deployment configs (Docker Compose quick-start, Helm chart values, CI backend config, task-sdk integration tests, a logging/multi-team doc) that would otherwise break for anyone following them as-is now that the postgres provider no longer bundles psycopg2-binary by default. (The same fix for providers/amazon's executor docs is split into a separate psycopg3-sync-amazon-docs PR, since it's outside airflow-core.)
• Newsfragment: airflow-core/newsfragments/69469.improvement.rst.

Test plan

• airflow-core/tests/unit/core/test_configuration.py::test_upgrade_postgres_metastore_conn updated to assert postgresql+psycopg, plus a case confirming an explicit postgresql+psycopg2:// value is left untouched.
• airflow-core/tests/unit/core/test_configuration.py::test_upgrade_postgres_metastore_conn_without_psycopg3: new test mocking find_spec absent, confirming the fallback to postgresql+psycopg2.
• airflow-core/tests/unit/core/test_sqlalchemy_config.py: new parametrized test covering all four combinations of {postgresql+psycopg2, postgresql+psycopg} × {psycopg3 available, not available}, confirming the executemany tuning follows the configured scheme rather than package importability.
• Verified behaviorally against bare/explicit-psycopg2/explicit-psycopg sql_alchemy_conn values with no live Postgres required.
• The two migration fixes are exercised by the existing full-alembic-chain CI jobs (airflow db reset && airflow db migrate --to-revision heads, run by every Postgres-backed core/integration/special-test job) — these were the actual jobs failing on CI before this fix. Additionally verified locally: a full db migrate --to-revision heads → db downgrade → db migrate --to-revision heads cycle against a live breeze Postgres, with postgresql+psycopg as the driver, completes cleanly.
• The backend-postgres.yml revert was verified by reproducing the exact failing "Migration Tests" CI job locally (breeze shell --backend postgres --mount-sources skip --python 3.10 --use-airflow-version 2.11.0 'airflow db reset --skip-init -y && airflow db migrate --to-revision heads'), which failed with NoSuchModuleError before the fix and completes cleanly (Database migrating done!) after it.
• test_serialized_dag.py::test_get_dependencies, test_dependencies.py::TestGetDependencies::test_should_response_200, and test_exceptions.py::TestUniqueConstraintErrorHandler verified against a live breeze Postgres (postgresql+psycopg driver) — all failed before these last two fixes and pass after.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Sonnet 5 following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.