Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion blog/20251005-postgres-marathon-2-001.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-001: Lightweight and heavyweight locks"
date: 2025-10-05
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
---

To warm up, let's talk about lightweight and heavyweight locks (or "regular locks" or just "locks").
Expand Down
2 changes: 1 addition & 1 deletion blog/20251005-postgres-marathon-2-002.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-002: Relation-level locks"
date: 2025-10-06
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
---

Let's talk about relation-level locks and various confusions, surprises and what is worth to remember in practice.
Expand Down
4 changes: 2 additions & 2 deletions blog/20251007-postgres-marathon-2-003.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-003: The roots of LWLock:LockManager"
date: 2025-10-07
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
image: /assets/blog/20251007-postgres-marathon-2-003.png
---

Expand Down Expand Up @@ -33,7 +33,7 @@ Because of this, fast, high-frequency queries (e.g. PK lookups) on multi-core sy

In 2011, Robert Haas implemented an additional optimization called fast-path locking, specifically designed for such cases ([commit](https://github.com/postgres/postgres/commit/3cba8999b343648c4c528432ab3d51400194e93b), it was released in 9.2 (2012). Fast-path locks are stored separately, individually for each backend, and protected by per-backend LWLocks, so contention doesn't happen. Before Postgres 18, no more than 16 ([`FP_LOCK_SLOTS_PER_BACKEND`](https://github.com/postgres/postgres/blob/1c4671f7b7c378cc26d1953785a3aa249152958b/src/include/storage/proc.h#L80-L86)) locks could be stored in this way. This mechanism can be applied only for AccessShareLock, RowShareLock, and RowExclusiveLock. The locks that are processed in this way can be observed in `pg_locks` (`fastpath=true`).

This helped a lot to improve performance of SELECTs that deal with no more than 16 relations (e.g., a PK lookup on a table with no more than 15 indexes), but in recent years, especially because of partitioning, it was once again not enough, so in Postgres 18 (2025) one more optimization was implemented.
This helped a lot to improve performance of SELECTs that deal with no more than 16 relations (e.g., a PK lookup on a table with no more than 15 indexes), but in recent years, especially because of partitioning, it was once again not enough, so in Postgres 18 (2025) one more optimization was implemented ([commit `c4d5cb71d`](https://github.com/postgres/postgres/commit/c4d5cb71d)).

// to be continued: we'll discuss PG18 and then entertaining benchmarks

4 changes: 2 additions & 2 deletions blog/20251008-postgres-marathon-2-004.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-004: Fast-path locking explained"
date: 2025-10-08
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
image: /assets/blog/20251008-postgres-marathon-2-004-cover.png
---

Expand Down Expand Up @@ -34,7 +34,7 @@ With partitioning becoming popular, 16 slots often isn't enough. A query touchin

This became a real problem for multiple PostgresAI clients in 2023 - they were experiencing severe LWLock:LockManager contention. We had some tests where we changed `FP_LOCK_SLOTS_PER_BACKEND` and confirmed that it mitigates LWLock:LockManager contention in certain cases, so [I proposed to make this hard-coded value configurable](https://www.postgresql.org/message-id/flat/CAM527d-uDn5osa6QPKxHAC6srOfBH3M8iXUM%3DewqHV6n%3Dw1u8Q%40mail.gmail.com). [Tomas Vondra](https://www.linkedin.com/in/tomasvondra/) was the first to respond, and it was clear he had also been studying similar cases.

In Postgres 18, the storage for fast-path locks has changed, now fast-path locks are stored in variable-sized arrays in separate shared memory (referenced via pointers from PGPROC). This allows variable sizing, so allowed number of fast-path locks for a backend scales with `max_locks_per_transaction` (default 64 slots). This is one of the trickiest parameters to understand fully, so we'll talk about it separately. Changing it requires Postgres restart.
In Postgres 18 ([commit `c4d5cb71d`](https://github.com/postgres/postgres/commit/c4d5cb71d)), the storage for fast-path locks has changed, now fast-path locks are stored in variable-sized arrays in separate shared memory (referenced via pointers from PGPROC). This allows variable sizing, so allowed number of fast-path locks for a backend scales with `max_locks_per_transaction` (default 64 slots). This is one of the trickiest parameters to understand fully, so we'll talk about it separately. Changing it requires Postgres restart.

To wrap a today, let's look at some benchmarks. These benchmarks were conducted by [Denis Morozov](https://www.linkedin.com/in/denis-morozov-5b92691/) from the PostgresAI team by request from GitLab – see [this GitLab issue](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/594#note_2786838563) (kudos to GitLab for keeping a lot of work publicly available, for wide open source community benefits – including Postgres community!)

Expand Down
4 changes: 2 additions & 2 deletions blog/20251009-postgres-marathon-2-005.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
title: "#PostgresMarathon 2-005: More LWLock:LockManager benchmarks for Postgres 18"
date: 2025-10-09
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
image: /assets/blog/20251009-postgres-marathon-2-005-cover.jpeg
---

In 2023-2024, after incidents that multiple customers of PostgresAI experienced, when production nodes were down because of LWLock:LockManager contention, we studied it in synthetic environments.

At that time, we managed to reproduce the issue only on large machines – ~100 or more vCPUs.

With PG18 release, this question started to bother me again: can we experience LWLock:LockManager on smaller machines?
With PG18 release (and its fast-path locking improvement, [commit `c4d5cb71d`](https://github.com/postgres/postgres/commit/c4d5cb71d)), this question started to bother me again: can we experience LWLock:LockManager on smaller machines?

[Denis Morozov](https://www.linkedin.com/in/denis-morozov-5b92691/) just published [results of benchmarks that successfully reproduce LWLock:LockManager contention in PG18 on 16-vCPU VMs](https://gitlab.com/postgres-ai/postgresql-consulting/tests-and-benchmarks/-/issues/69).

Expand Down
2 changes: 1 addition & 1 deletion blog/20251010-postgres-marathon-2-006.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-006: Mysterious max_locks_per_transaction"
date: 2025-10-10
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager]
image: /assets/blog/20251010-postgres-marathon-2-006-cover.png
---

Expand Down
2 changes: 1 addition & 1 deletion blog/20251013-postgres-marathon-2-007.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-007: Should we worry about pg_blocking_pids()'s observer effect?"
date: 2025-10-13
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks, monitoring]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager, monitoring]
image: /assets/blog/20251013-postgres-marathon-2-007-cover.jpg
---

Expand Down
2 changes: 1 addition & 1 deletion blog/20251014-postgres-marathon-2-008.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-008: LWLock:LockManager and prepared statements"
date: 2025-10-14 23:59:59
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks, prepared statements]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager, prepared statements]
image: /assets/blog/20251014-postgres-marathon-2-008-cover.jpg
---

Expand Down
2 changes: 1 addition & 1 deletion blog/20251028-postgres-marathon-2-009.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-009: Prepared statements and partitioned table lock explosion, part 1"
date: 2025-10-28 12:00:00
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks, prepared statements, partitioning]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager, prepared statements, partitioning]
---

In [#PostgresMarathon 2-008](https://postgres.ai/blog/20251014-postgres-marathon-2-008), we discovered that prepared statements can dramatically reduce `LWLock:LockManager` contention by switching from planner locks (which lock everything) to executor locks (which lock only what's actually used). Starting with execution 7, we saw locks drop from 6 (table + 5 indexes) to just 1 (table only).
Expand Down
2 changes: 1 addition & 1 deletion blog/20251029-postgres-marathon-2-010.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-010: Prepared statements and partitioned table lock explosion, part 2"
date: 2025-10-29 23:59:59
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks, prepared statements, partitioning]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager, prepared statements, partitioning]
---

In [#PostgresMarathon 2-009](https://postgres.ai/blog/20251028-postgres-marathon-2-009), we focused on Lock Manager's behavior when dealing with prepared statements and partitioned tables.
Expand Down
2 changes: 1 addition & 1 deletion blog/20251030-postgres-marathon-2-011.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "#PostgresMarathon 2-011: Prepared statements and partitioned tables — the paradox, part 3"
date: 2025-10-30 23:59:59
authors: nik
tags: [Postgres insights, PostgresMarathon, internals, locks, prepared statements, partitioning]
tags: [Postgres insights, PostgresMarathon, internals, locks, lockmanager, prepared statements, partitioning]
---

In [#PostgresMarathon 2-009](https://postgres.ai/blog/20251028-postgres-marathon-2-009) and [#PostgresMarathon 2-010](https://postgres.ai/blog/20251029-postgres-marathon-2-010), we explored why execution 6 causes a lock explosion when building a generic plan for partitioned tables — the planner must lock all 52 relations because it can't prune without parameter values.
Expand Down
Loading