Skip to content

server/db/pg: Add DB foundation for mesh - #3628

Open
martonp wants to merge 16 commits into
decred:masterfrom
martonp:mesh-db-foundation
Open

martonp wants to merge 16 commits into
decred:masterfrom
martonp:mesh-db-foundation

Conversation

@martonp

@martonp martonp commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

This PR builds on #3627 and adds PostgreSQL support for mesh replication, including the event log, snapshot export and restore, and the v9 schema upgrade.

server/db/pg: Add event log with atomic state updates

Create the event log table to store each event's sequence number, kind,
payload, transaction data, and tip hash.

Add applyEventTx to run an event's database changes and append its log
entry in one transaction. Both are committed together, and a mismatch
with a supplied sequence number or expected tip hash rolls back the
changes. This helper is for use by event appliers.

Add EventLogFrontier and EventLogEntriesAfter so the mesh package can
query the event log table.

server/db/pg: Classify tables and support clearing event-sourced state

Define which tables are included in snapshots, which must be empty
before loading a snapshot, and which are preserved as local configuration.
Check at startup that every table in public and the stored market schemas
has a classification to avoid future developers adding tables without
considering how snapshotting will be affected.

Add HasNoEventSourcedState to check whether all event-sourced data is
absent, as required before loading a snapshot. Add WipeEventSourcedState
to clear all event-sourced data before restoring from a peer.

server/db/pg: Add snapshot export and restore

Add WriteSnapshot, used by the master to create a snapshot that is
sent to the slave, and also add LoadSnapshot, used by the slave to
populate its DB with the master's data.

The snapshot includes:

  • The event-log sequence number and tip hash corresponding to the snapshot.
  • All accounts, bonds, prepaid bonds, and reputation points.

For configured markets, it also includes:

  • Market lifecycle state.
  • Active orders, cancels, and matches.
  • The past 30 days of archived orders, archived cancels, and matches, plus
    older archived orders referenced by included matches or archived cancels.
  • The latest 1,000 candles from each candle table.
  • Recent epoch reports needed to rebuild unfinished candles.

server/db/pg: Add v9 schema upgrade for mesh replication

Add the v9 upgrade and update table schemas. The upgrade makes the
following changes:

  • Create the event_log table.
  • Create the market_lifecycle table to store market state, scheduled
    suspensions and resumptions, epoch progress, and trading parameters.
  • Remove the legacy accounts.fee_asset column.
  • Remove commitment and preimage uniqueness constraints from
    orders_archived and cancels_archived. Nodes may retain different
    archived histories, so these constraints could make archiving the same
    order succeed on one node and fail on another.
  • Add non-unique commitment indexes to orders_archived and
    cancels_archived for lookups.
  • Add indexes on makerOrder and takerOrder for active matches, to
    support finding an order's active matches during future swap processing.
  • Add a genesis event when a database has existing event-sourced data and
    an empty event log. It includes a random nonce, so each legacy database
    initialized this way has a distinct history. Databases with no
    event-sourced data do not receive a genesis event.

Create the event log table to store each event's sequence number, kind,
payload, transaction data, and tip hash.

Add `applyEventTx` to run an event's database changes and append its log
entry in one transaction. Both are committed together, and a mismatch
with a supplied sequence number or expected tip hash rolls back the
changes. This helper is for use by event appliers.

Add `EventLogFrontier` and `EventLogEntriesAfter` so the mesh package can
query the event log table.
Define which tables are included in snapshots, which must be empty
before loading a snapshot, and which are preserved as local configuration.
Check at startup that every table in `public` and the stored market schemas
has a classification to avoid future developers adding tables without
considering how snapshotting will be affected.

Add `HasNoEventSourcedState` to check whether all event-sourced data is
absent, as required before loading a snapshot. Add `WipeEventSourcedState`
to clear all event-sourced data before restoring from a peer.
Add `WriteSnapshot`, used by the master to create a snapshot that is
sent to the slave, and also add `LoadSnapshot`, used by the slave to
populate its DB with the master's data.

The snapshot includes:

- The event-log sequence number and tip hash corresponding to the snapshot.
- All accounts, bonds, prepaid bonds, and reputation points.

For configured markets, it also includes:

- Market lifecycle state.
- Active orders, cancels, and matches.
- The past 30 days of archived orders, archived cancels, and matches, plus
  older archived orders referenced by included matches or archived cancels.
- The latest 1,000 candles from each candle table.
- Recent epoch reports needed to rebuild unfinished candles.
Add the v9 upgrade and update table schemas. The upgrade makes the
following changes:

- Create the `event_log` table.
- Create the `market_lifecycle` table to store market state, scheduled
  suspensions and resumptions, epoch progress, and trading parameters.
- Remove the legacy `accounts.fee_asset` column.
- Remove commitment and preimage uniqueness constraints from
  `orders_archived` and `cancels_archived`. Nodes may retain different
  archived histories, so these constraints could make archiving the same
  order succeed on one node and fail on another.
- Add non-unique commitment indexes to `orders_archived` and
  `cancels_archived` for lookups.
- Add indexes on `makerOrder` and `takerOrder` for active matches, to
  support finding an order's active matches during future swap processing.
- Add a genesis event when a database has existing event-sourced data and
  an empty event log. It includes a random nonce, so each legacy database
  initialized this way has a distinct history. Databases with no
  event-sourced data do not receive a genesis event.
@martonp
martonp force-pushed the mesh-db-foundation branch 3 times, most recently from cee820a to e391aa3 Compare September 22, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant