Repository navigation
PBS-64: Implement incrementing buffer size strategy in sender_context::get_event() - #200
Merged
Merged
Conversation
…:get_event() https://perconadev.atlassian.net/browse/PBS-64 Problem: The sender fetches binlog bytes from storage in fixed-size blocks (default 1 MiB). When the first event starting at the block boundary is itself larger than the block, zero complete events parse out of the fetched buffer and the sender returns a hard error. From that point on the sender can never progress past any event larger than one block. Solution: Before parsing the fetched buffer, peek the first event's advertised size from its common header and compare it against the number of bytes actually received. When the event does not fit, re-issue a single fetch at the same offset with length set to that exact size; the parse then succeeds with one complete event. The common header is always the first 19 bytes of a well-formed event (matches "default_common_header_length" and the "size_in_bytes" invariant on "common_header_view_base"). A fetched buffer shorter than that, or an exact-size retry that still yields no complete event, both indicate a corrupt or truncated binlog and are reported as errors with no further retry. The advertised size is bounded by a new "max_event_size_bytes" constant set to 1 GiB, matching MySQL's "MAX_MAX_ALLOWED_PACKET" and "replica_max_allowed_packet" ceiling. A corrupt or malicious header claiming a multi-GiB event is rejected before any allocation is made. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
https://perconadev.atlassian.net/browse/PBS-64 New MTR test "rs_large_event" exercises the full source -> PBS -> consumer pipeline with a single row-based event large enough to exceed PBS's fetch block. A LONGBLOB row of ~1.5 MiB is inserted on the source, PBS fetches the resulting binlogs into its storage, and mysqlbinlog connects to the PBS replication-source listener and drains the event back out. The test then (a) waits for the "sender : block too small for first event" entry in the PBS log, which only appears when sender_context detected that the first event in a block did not fit and re-issued the fetch at exact size, and (b) replays the mysqlbinlog dump into a backup database and verifies the restored row matches the original by (id, length, MD5). The MD5 comparison is used in place of diff_tables.inc because the default sort buffer cannot hold a 1.5 MiB blob and would make the test fail for a reason unrelated to what is being covered. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.