PBS-59 Reconcile storage against binlog index on startup

README.md

@@ -629,6 +629,22 @@ If this an optional section that specifies keyring configuration parameters. It
   - 'h' (e.g. "42h") means hours ('42 * 60 * 60' seconds)
   - 'd' (e.g. "42d") means days ('42 * 60 * 60 *24' seconds)
 
+##### Storage consistency checks
+
+When the Binlog Server process is not properly shut down (killed, crashed, power loss), the storage may be left in an inconsistent state. Every time the storage is opened, the following hierarchy of sources of truth is used to check it:
+1. `binlog.index` defines the set of binlog files in the storage. Only objects that are known to be left after an interrupted operation of the Binlog Server are treated as garbage:
+   - the binlog data file (and its metadata file) that follows the last binlog file listed in `binlog.index` - left after an interrupted binlog rotation;
+   - binlog data / metadata files with the same base name as the first binlog file listed in `binlog.index` and a smaller sequence number - left after an interrupted purge;
+   - if `binlog.index` does not exist, a single binlog data file (and its metadata file) - left after an interrupted creation of the very first binlog file;
+   - temporary objects (`*.tmp`) left after interrupted writes.
+
+   Any other object not listed in `binlog.index` (including objects not created by the Binlog Server) makes the storage inconsistent.
+2. The binlog metadata file (`<binlog name>.json`) defines the size of the binlog data file. If the most recent binlog data file is bigger than recorded in its metadata, it is truncated to the recorded size.
+
+In the `fetch`, `pull` and `purge_binlogs` modes, the storage is fixed accordingly (garbage objects are removed after the whole storage has been checked) and the operation continues. The operation fails if the storage cannot be fixed automatically: a binlog file listed in `binlog.index` has no data file or no valid metadata file, a binlog data file is smaller than recorded in its metadata, the size mismatch is found in a binlog file that is not the most recent one, the storage contains an object not listed in `binlog.index` that is not garbage, or `binlog.index` is missing while the storage contains anything but a single binlog file. Nothing is removed from a storage that cannot be fixed. Please notice that these fixes are not safe to perform while another instance is fetching / pulling data to the same storage (see the note about the `purge_binlogs` mode above).
+
+In the `list`, `search_by_timestamp` and `search_by_gtid_set` modes, the storage is never modified, and any inconsistency found is ignored: these modes typically run as a second, read-only process just to inspect the storage, so the JSON response reflects whatever is on disk at the time, as-is (if `binlog.index` is missing while the storage contains anything but a single binlog file, the storage is considered empty).
+
 ##### Storage URI format
 
 - When `<storage.backend>` is set to `file`, `<storage.uri>` must be `file://...`.

mtr/binlog_streaming/include/assert_storage_object_absent.inc

@@ -0,0 +1,39 @@
+#
+# Asserts that an object does not exist in the backend storage.
+#
+# Usage:
+#   --let $storage_backend = file
+#   --let $storage_object = $MYSQL_TMP_DIR/first
+#   --source assert_storage_object_absent.inc
+#
+# or
+#   --let $aws_cli = AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... aws
+#   --let $storage_backend = s3
+#   --let $aws_s3_bucket = my-bucket
+#   --let $storage_object = /vault/first
+#   --source assert_storage_object_absent.inc
+#
+# $storage_backend - determines storage backend type (either 'file' or 's3')
+# $aws_cli         - path to AWS command line interface (cli) tools with AWS_ACCESS_KEY_ID /
+#                    AWS_SECRET_ACCESS_KEY environment variables set appropriately
+#                    (needed only if $storage_backend is 's3')
+# $aws_s3_bucket   - AWS S3 bucket name (needed only if $storage_backend is 's3')
+# $storage_object  - if $storage_backend is 'file', a path to the file on a
+#                    local filesystem; if $storage_backend is 's3', a path to
+#                    the object on AWS S3
+#
+
+if ($storage_backend == file)
+{
+  --error 1
+  --file_exists $storage_object
+}
+if ($storage_backend == s3)
+{
+  # 'aws s3 ls' cannot be used here as it matches objects by prefix;
+  # 'aws s3api head-object' fails with exit code 254 (AWS CLI v2) or
+  # 255 (AWS CLI v1) when the object does not exist
+  --let $assert_storage_object_absent_key = `SELECT TRIM(LEADING '/' FROM '$storage_object')`
+  --error 254,255
+  --exec $aws_cli s3api head-object --bucket $aws_s3_bucket --key $assert_storage_object_absent_key > /dev/null 2>&1
+}

mtr/binlog_streaming/include/download_storage_object.inc

@@ -0,0 +1,37 @@
+#
+# Copies an object from the backend storage to a file on a local filesystem.
+#
+# Usage:
+#   --let $storage_backend = file
+#   --let $storage_object = $MYSQL_TMP_DIR/first
+#   --let $local_file = $MYSQL_TMP_DIR/second
+#   --source download_storage_object.inc
+#
+# or
+#   --let $aws_cli = AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... aws
+#   --let $storage_backend = s3
+#   --let $aws_s3_bucket = my-bucket
+#   --let $storage_object = /vault/first
+#   --let $local_file = $MYSQL_TMP_DIR/second
+#   --source download_storage_object.inc
+#
+# $storage_backend - determines storage backend type (either 'file' or 's3')
+# $aws_cli         - path to AWS command line interface (cli) tools with AWS_ACCESS_KEY_ID /
+#                    AWS_SECRET_ACCESS_KEY environment variables set appropriately
+#                    (needed only if $storage_backend is 's3')
+# $aws_s3_bucket   - AWS S3 bucket name (needed only if $storage_backend is 's3')
+# $storage_object  - if $storage_backend is 'file', a path to the source file on
+#                    a local filesystem; if $storage_backend is 's3', a path to
+#                    the source object on AWS S3
+# $local_file      - a path to the destination file on a local filesystem
+#                    (must not exist)
+#
+
+if ($storage_backend == file)
+{
+  --copy_file $storage_object $local_file
+}
+if ($storage_backend == s3)
+{
+  --exec $aws_cli s3 cp s3://$aws_s3_bucket$storage_object $local_file > /dev/null
+}

mtr/binlog_streaming/include/remove_storage_object.inc

@@ -0,0 +1,33 @@
+#
+# Removes an object from the backend storage.
+#
+# Usage:
+#   --let $storage_backend = file
+#   --let $storage_object = $MYSQL_TMP_DIR/first
+#   --source remove_storage_object.inc
+#
+# or
+#   --let $aws_cli = AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... aws
+#   --let $storage_backend = s3
+#   --let $aws_s3_bucket = my-bucket
+#   --let $storage_object = /vault/first
+#   --source remove_storage_object.inc
+#
+# $storage_backend - determines storage backend type (either 'file' or 's3')
+# $aws_cli         - path to AWS command line interface (cli) tools with AWS_ACCESS_KEY_ID /
+#                    AWS_SECRET_ACCESS_KEY environment variables set appropriately
+#                    (needed only if $storage_backend is 's3')
+# $aws_s3_bucket   - AWS S3 bucket name (needed only if $storage_backend is 's3')
+# $storage_object  - if $storage_backend is 'file', a path to the file on a
+#                    local filesystem; if $storage_backend is 's3', a path to
+#                    the object on AWS S3
+#
+
+if ($storage_backend == file)
+{
+  --remove_file $storage_object
+}
+if ($storage_backend == s3)
+{
+  --exec $aws_cli s3 rm s3://$aws_s3_bucket$storage_object > /dev/null
+}

mtr/binlog_streaming/include/upload_storage_object.inc

@@ -0,0 +1,41 @@
+#
+# Copies a file from a local filesystem to the backend storage (overwriting
+# the destination object if it already exists).
+#
+# Usage:
+#   --let $storage_backend = file
+#   --let $local_file = $MYSQL_TMP_DIR/first
+#   --let $storage_object = $MYSQL_TMP_DIR/second
+#   --source upload_storage_object.inc
+#
+# or
+#   --let $aws_cli = AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... aws
+#   --let $storage_backend = s3
+#   --let $local_file = $MYSQL_TMP_DIR/first
+#   --let $aws_s3_bucket = my-bucket
+#   --let $storage_object = /vault/second
+#   --source upload_storage_object.inc
+#
+# $storage_backend - determines storage backend type (either 'file' or 's3')
+# $aws_cli         - path to AWS command line interface (cli) tools with AWS_ACCESS_KEY_ID /
+#                    AWS_SECRET_ACCESS_KEY environment variables set appropriately
+#                    (needed only if $storage_backend is 's3')
+# $local_file      - a path to the source file on a local filesystem
+# $aws_s3_bucket   - AWS S3 bucket name (needed only if $storage_backend is 's3')
+# $storage_object  - if $storage_backend is 'file', a path to the destination
+#                    file on a local filesystem; if $storage_backend is 's3', a
+#                    path to the destination object on AWS S3
+#
+
+if ($storage_backend == file)
+{
+  # '--copy_file' refuses to overwrite an existing file, hence removing the
+  # destination first (ignoring the case when it does not exist yet)
+  --error 0,1
+  --remove_file $storage_object
+  --copy_file $local_file $storage_object
+}
+if ($storage_backend == s3)
+{
+  --exec $aws_cli s3 cp $local_file s3://$aws_s3_bucket$storage_object > /dev/null
+}

mtr/binlog_streaming/r/storage_reconciliation.result

@@ -0,0 +1,171 @@
+*** Resetting replication at the very beginning of the test.
+
+*** Generating two binary logs with some data in each of them.
+CREATE TABLE t1(id INT UNSIGNED NOT NULL AUTO_INCREMENT, PRIMARY KEY(id)) ENGINE=InnoDB;
+INSERT INTO t1 VALUES(DEFAULT);
+FLUSH BINARY LOGS;
+INSERT INTO t1 VALUES(DEFAULT);
+
+*** Generating a configuration file in JSON format for the Binlog
+*** Server utility.
+
+*** Determining binlog file directory from the server.
+
+*** Creating a temporary directory <BINSRV_STORAGE_PATH> for storing
+*** binlog files downloaded via the Binlog Server utility.
+
+*** Executing the Binlog Server utility to download all binlog data
+*** from the server.
+
+*** Saving a copy of the second binlog data and metadata files to be
+*** used for simulating garbage objects.
+
+*** 1. Binlog files left after an interrupted binlog rotation are
+***    removed; querying-only operations ignore them without
+***    modifying the storage.
+
+*** Simulating a binlog rotation interrupted after the new binlog
+*** metadata was written but before the binlog index was updated, and
+*** an interrupted binlog index update.
+
+*** Executing the Binlog Server utility in the 'list' mode and
+*** expecting it to ignore the garbage objects without modifying the
+*** storage.
+include/assert_grep.inc [List response is not affected]
+
+*** Executing the Binlog Server utility and expecting it to remove
+*** garbage objects.
+include/assert_grep.inc [Found orphaned binlog data file]
+include/assert_grep.inc [Found orphaned binlog metadata file]
+include/assert_grep.inc [Removed orphaned objects]
+include/assert_grep.inc [Removed temporary object]
+
+*** 2. The most recent binlog data file is truncated to the size from
+***    its metadata; querying-only operations ignore the size mismatch
+***    without modifying the storage.
+
+*** Appending extra bytes to the second binlog data file.
+
+*** Executing the Binlog Server utility in the 'list' mode and
+*** expecting it to ignore the size mismatch without modifying the
+*** storage.
+include/assert_grep.inc [List response is not affected]
+
+*** Executing the Binlog Server utility and expecting it to truncate
+*** the second binlog data file.
+include/assert_grep.inc [Truncated the second binlog data file]
+
+*** 3. The storage with a missing metadata file of a binlog listed in
+***    the binlog index, with a size mismatch in an older binlog file or
+***    with an object that was not created by the Binlog Server cannot
+***    be repaired, and garbage objects are not removed from it.
+
+*** Simulating an interrupted binlog rotation one more time.
+
+*** Removing the first binlog metadata file.
+
+*** Executing the Binlog Server utility and expecting it to fail.
+include/assert_grep.inc [Refused to open storage with missing binlog metadata]
+include/assert_grep.inc [Nothing removed from the storage with missing binlog metadata]
+
+*** Restoring the first binlog metadata file.
+
+*** Appending extra bytes to the first (not the most recent) binlog
+*** data file.
+
+*** Executing the Binlog Server utility and expecting it to fail.
+include/assert_grep.inc [Refused to open storage with size mismatch in an older binlog]
+include/assert_grep.inc [Nothing removed from the storage with size mismatch in an older binlog]
+
+*** Restoring the first binlog data file.
+
+*** Adding an object that was not created by the Binlog Server.
+
+*** Executing the Binlog Server utility and expecting it to fail.
+include/assert_grep.inc [Refused to open storage with a foreign object]
+include/assert_grep.inc [Nothing removed from the storage with a foreign object]
+
+*** Removing the foreign object and the garbage objects.
+
+*** 4. The storage with binlog files but without the binlog index cannot
+***    be repaired; querying-only operations just ignore it.
+
+*** Removing the binlog index.
+
+*** Executing the Binlog Server utility in the 'list' mode and
+*** expecting an empty response.
+include/assert_grep.inc [List response contains no binlog files]
+
+*** Executing the Binlog Server utility and expecting it to fail.
+include/assert_grep.inc [Refused to open storage without binlog index]
+include/assert_grep.inc [Nothing removed from the storage without binlog index]
+
+*** Restoring the binlog index.
+
+*** Flushing the second binary log and dropping the table.
+FLUSH BINARY LOGS;
+DROP TABLE t1;
+
+*** Executing the Binlog Server utility to receive the rest of the
+*** events.
+
+*** 5. The 'purge_binlogs' operation removes garbage objects left after
+***    an improper shutdown as well.
+
+*** Saving a copy of the first binlog data and metadata files to be
+*** used for simulating an interrupted binlog purge.
+
+*** Simulating an interrupted binlog rotation one more time (the third
+*** binlog file is already in the storage now).
+
+*** Executing the Binlog Server utility in the 'purge_binlogs' mode.
+include/assert_grep.inc [Purge response is not affected]
+
+*** 6. Binlog files left after an interrupted binlog purge are removed.
+
+*** Simulating a binlog purge interrupted after the binlog index was
+*** updated but before the purged binlog files were removed.
+
+*** Executing the Binlog Server utility and expecting it to remove
+*** garbage objects.
+include/assert_grep.inc [Found purged binlog data file]
+include/assert_grep.inc [Removed purged binlog files]
+
+*** 7. Querying-only operations do not crash even if the metadata of
+***    every binlog file listed in the binlog index is missing (PBS-79).
+
+*** Removing the metadata files of all remaining binlog files.
+
+*** Executing the Binlog Server utility in the 'list' mode and
+*** expecting it to exit cleanly with an empty response.
+include/assert_grep.inc [List response contains no binlog files]
+
+*** 8. Querying-only operations ignore, and streaming operations
+***    recover, a storage left after an interrupted creation of the
+***    first binlog file (no binlog index, at most one binlog's data
+***    and metadata).
+
+*** Removing the remaining binlog data files and the binlog index.
+
+*** Simulating an interrupted creation of the first binlog file after
+*** both its data and metadata were written.
+
+*** Executing the Binlog Server utility in the 'list' mode and
+*** expecting it to ignore the leftover objects without modifying the
+*** storage.
+include/assert_grep.inc [List response contains no binlog files]
+
+*** Removing the metadata file to simulate an interrupted creation of
+*** the first binlog file after only its data was written.
+
+*** Executing the Binlog Server utility and expecting it to recover by
+*** removing the leftover binlog data file.
+include/assert_grep.inc [Found orphaned first binlog data file]
+include/assert_grep.inc [No orphaned first binlog metadata file found]
+include/assert_grep.inc [Removed orphaned first binlog file]
+
+*** Removing the Binlog Server utility storage directory.
+
+*** Removing the Binlog Server utility log file.
+
+*** Removing the Binlog Server utility configuration file.

mtr/binlog_streaming/t/storage_reconciliation.combinations

@@ -0,0 +1,5 @@
+[plain]
+init-connect = SET @binsrv_encryption = 'off'
+
+[encrypted]
+init-connect = SET @binsrv_encryption = 'on'