PBS-59 Reconcile storage against binlog index on startup by lukin-oleksiy · Pull Request #195 · Percona-Lab/percona-binlog-server

lukin-oleksiy · 2026-09-30T10:54:03Z

Problem

A hard kill between writing a new binlog file's metadata and updating
binlog.index left the storage unable to start ("storage contains an object
that is not referenced in the binlog index"). Interrupted purges had the same
problem.

Solution

When the storage is opened, it is reconciled top-down:

binlog.index is the source of truth for the set of binlog files.
Binlog data / metadata files not listed in it (left after an interrupted
binlog file creation or an interrupted purge) and *.tmp objects left
after interrupted writes are garbage and are removed. A missing binlog
index means the storage has no binlog files, unless it contains more than
a single binlog file, which is an error (to avoid wiping the storage when
the index is lost).
Binlog metadata (<binlog>.json) is the source of truth for the data
file size. The most recent binlog data file is truncated to the recorded
size. This also works for encrypted files: AES-CTR keeps byte offsets, and
encryption resumes from an arbitrary (not block-aligned) offset.

Safety rules:

Only objects whose names have the form of objects created by the Binlog
Server (metadata.json, binlog.index, <base>.<seq>, <base>.<seq>.json
and their .tmp versions) are considered. Any other object (e.g. nested S3
keys, operator's files) is logged and left intact.
The storage is not opened if it cannot be made consistent: missing data
file or invalid metadata of an indexed binlog, data file smaller than
recorded, size mismatch in a non-latest binlog, lost index.
Garbage is removed only after the whole storage has been validated, so
nothing is removed from a storage that cannot be repaired.
Running purge_binlogs while another instance is fetching / pulling to the
same storage remains unsupported (as already documented in README); the
repairs described above are not safe in that situation either.

Reporting

fetch / pull: every fix is logged with the warning severity.
purge_binlogs: appends its messages to the configured log file (which may
be used by a running fetch / pull) with the [storage-maintenance] tag;
prints to stderr if no log file is configured.
Log files are now always written in the append mode (after truncation for
fetch / pull), so messages from both processes do not overwrite each
other.
list / search_by_timestamp / search_by_gtid_set: never modify the
storage; problems and how they will be fixed are printed to stderr at the
configured logging level, binlog files that cannot be used are skipped.
Nothing is printed for a consistent storage.
JSON responses of all operations are unchanged.

Other changes

cout_logger is generalized into ostream_logger (stdout / stderr),
logger_factory::create() gets creation options, unused null_logger is
removed.
The filesystem backend fsyncs stream data before returning, so binlog
metadata never describes bytes that are not on disk.
Build (1st commit, all CMakeLists.txt changes): Boost lookup via
BoostConfig.cmake with a Boost::asio fallback for distro packages; AWS SDK
CPP fallback lookup in a single ../aws-sdk-cpp-install-<preset> directory
next to the source tree (several such directories are an error); updated
source list.

Testing

New storage_reconciliation MTR test with plain and encrypted
combinations: garbage removal, foreign objects kept, querying-only
reporting, truncation (and resumed encryption at an unaligned offset,
verified by decrypting and comparing with the server's binlogs),
unrepairable storages with nothing removed, purge_binlogs logging.
Backend-agnostic storage object helper includes for MTR.
Full binlog_streaming suite (41 tests, --big-test) passes on the file
backend; the S3 backend was not tested.
Unit tests pass; clang-tidy and clang-format clean.

🤖 Generated with Claude Code

Copilot

Copilot review overview

🟡 Changes recommended

Ordinal validation can silently accept a missing leading binlog instead of reporting storage corruption.

Review effort: Balanced
Findings: 1

Open (1)

Missing purge-horizon validation allows lost binlog omission · New

What changed in this PR

Makes binlog metadata the storage source of truth and adds crash-time reconciliation.

Changes:

Adds ordinals, purge horizons, and derived index regeneration.
Adds durable filesystem writes and reconciliation tests/helpers.
Updates Boost and AWS SDK discovery.

File	Description
`src/binsrv/storage_metadata.hpp`	Adds purge horizon metadata.
`src/binsrv/storage_metadata.cpp`	Initializes the new field.
`src/binsrv/storage_metadata_fwd.hpp`	Bumps metadata version.
`src/binsrv/storage_core.hpp`	Defines reconciliation interfaces and state.
`src/binsrv/storage_core.cpp`	Implements recovery, purge, and index derivation.
`src/binsrv/filesystem_storage_backend.hpp`	Tracks the active stream path.
`src/binsrv/filesystem_storage_backend.cpp`	Fsyncs streamed data.
`src/binsrv/binlog_file_metadata.hpp`	Adds binlog ordinals.
`src/binsrv/binlog_file_metadata.cpp`	Initializes ordinal metadata.
`src/binsrv/binlog_file_metadata_fwd.hpp`	Bumps binlog metadata version.
`mtr/binlog_streaming/t/storage_reconciliation.test`	Tests reconciliation scenarios.
`mtr/binlog_streaming/r/storage_reconciliation.result`	Adds expected test output.
`mtr/binlog_streaming/include/upload_storage_object.inc`	Adds backend-neutral uploads.
`mtr/binlog_streaming/include/remove_storage_object.inc`	Adds backend-neutral removal.
`mtr/binlog_streaming/include/download_storage_object.inc`	Adds backend-neutral downloads.
`mtr/binlog_streaming/include/assert_storage_object_absent.inc`	Adds absence assertions.
`CMakeLists.txt`	Adjusts dependency discovery.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  // 'records' is expected to be sorted by ordinal here, so any pair of
+  // adjacent records with non-consecutive ordinals indicates either a
+  // duplicate or a missing binlog metadata object
+  const auto gap_it{std::ranges::adjacent_find(
+      records, [](const binlog_record &previous, const binlog_record &next) {
+        return next.ordinal != previous.ordinal + 1ULL;
+      })};


- Use BoostConfig.cmake (CMP0167 NEW) instead of the deprecated FindBoost module and make the header-only 'asio' component optional, defining a Boost::asio fallback target on top of Boost::headers when the Boost installation (e.g. Debian / Ubuntu packages) does not provide one. - When AWS SDK CPP is not found in the standard locations (the CMake presets point CMAKE_PREFIX_PATH to the matching installation), also look for an installation made by the AWS SDK CPP presets next to the source tree ('../aws-sdk-cpp-install-<preset>'), so that a plain 'cmake' works without installing the SDK into a system prefix. Several such installations are reported as an error instead of picking an arbitrary one. - Source list: 'cout_logger' is replaced with 'ostream_logger', unused 'null_logger' is removed (the sources are changed in the next commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A hard kill between writing a new binlog file's metadata and updating 'binlog.index' left the storage unable to start ("storage contains an object that is not referenced in the binlog index"). Interrupted purges had the same problem. Storage objects are now reconciled top-down when the storage is opened: - 'binlog.index' is the source of truth for the set of binlog files. Binlog data / metadata files not referenced in it (left after an interrupted binlog file creation or an interrupted purge) and temporary objects are garbage. A missing binlog index means that the storage has no binlog files, unless the storage contains more than a single binlog file, which is an error. - Only objects whose names have the form of objects created by the Binlog Server are considered; any other object is logged and left intact. - Binlog metadata is the source of truth for the binlog data file size: the most recent binlog data file is truncated to the size recorded in its metadata (also for encrypted files, as CTR keeps byte offsets). - The storage is not opened if it cannot be made consistent: missing binlog data file or invalid metadata of an indexed binlog, binlog data file smaller than recorded, size mismatch in a non-latest binlog. Garbage is removed only after the whole storage has been validated, so nothing is removed from a storage that cannot be repaired. - 'fetch' / 'pull' log every fix. 'purge_binlogs' appends its messages to the configured log file marked with the '[storage-maintenance]' tag (or prints them to stderr if no log file is configured). Log files are now always written in the append mode (after truncation, unless the content must be kept), so that messages from both processes do not overwrite each other. - 'list' / 'search_by_timestamp' / 'search_by_gtid_set' never modify the storage; they print found problems and how they will be fixed to stderr, using the configured logging level, and skip binlog files that cannot be used. JSON responses are not changed. - 'cout_logger' is generalized into 'ostream_logger' (stdout / stderr), 'logger_factory::create()' gets creation options, unused 'null_logger' is removed. - The filesystem backend now fsyncs stream data before returning, so binlog metadata never describes bytes that are not on disk. - Added 'storage_reconciliation' MTR test (plain and encrypted combinations) and backend-agnostic storage object helper includes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lukin-oleksiy force-pushed the PBS-59-auto-recovery-signal11 branch 2 times, most recently from 5cec848 to 37a90ad Compare September 30, 2026 11:07

lukin-oleksiy requested review from percona-ysorokin and a balanced review from Copilot October 1, 2026 09:10

Copilot started reviewing on behalf of lukin-oleksiy October 1, 2026 09:11 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

lukin-oleksiy force-pushed the PBS-59-auto-recovery-signal11 branch from 37a90ad to fcd02de Compare October 8, 2026 10:30

lukin-oleksiy and others added 2 commits October 8, 2026 14:03

lukin-oleksiy force-pushed the PBS-59-auto-recovery-signal11 branch from fcd02de to 9f5e33e Compare October 8, 2026 11:04

lukin-oleksiy changed the title ~~PBS-59 auto recovery signal11~~ PBS-59 Reconcile storage against binlog index on startup Oct 8, 2026

Navigation Menu

Repository navigation

PBS-59 Reconcile storage against binlog index on startup - #195

PBS-59 Reconcile storage against binlog index on startup#195
lukin-oleksiy wants to merge 2 commits into
Percona-Lab:mainfrom
lukin-oleksiy:PBS-59-auto-recovery-signal11

lukin-oleksiy commented Sep 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lukin-oleksiy commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Reporting

Other changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lukin-oleksiy commented Sep 30, 2026 •

edited

Loading