storage: Fast approximate snapshot partitioning by peterdukelarsen · Pull Request #37994 · MaterializeInc/materialize

peterdukelarsen · 2026-07-31T23:14:35Z

Motivation

The boundary computation for parallel snapshotting was about as slow as the single-threaded snapshot in many cases, i.e.
Parallelized (~2h runtime):

Single-threaded (~1h50m runtime):

With this change we got rid of the boundary computation and were able to snapshot 1B rows in ~16m:

Follow this anchor link for full details on the specs/data: https://app.notion.com/p/materialize/MySQL-Parallel-Snapshot-Smoke-Test-3a613f48d37b806fa311d1bfe9d4ec4c?source=copy_link#3a913f48d37b80a3bbb2dd05f6bc94c2

Description

Probes the values of a single-column string primary key using EXPLAIN and SELECT LIKE queries to compute approximate partitioning.

In slightly more detail we:

Grab all of the unique first characters of the primary key strings and use EXPLAIN to estimate their row count
For any character with a high row count we will redo the process one character deeper
We continue stepping down into longer prefixes until we run out of a configured number of calls to make or we've resolved granular enough buckets to build partition boundaries
Finally, we pick partition boundaries based on the total estimated row count (which could diverge from the real row count or the regular estimated row count by a good bit), and just walk the partitions in order computing boundary keys, which are just prefixes.

Downsides

Complexity
Won't work in all cases -- particularly any cases with extremely high cardinality characters in their prefixes could devolve. To work around this we'll follow up with implementing some caps on the number of queries we make and/or the time we spend, but obviously that will just prevent regression not make this work for all collections of string PKs.

Verification

Deployed 777e7e8 to staging and snapshot 1B row table (August 3, around noon-3pm EDT)

Add a probe module exposing KeyProber over a string key column of a table: optimizer row count estimates for half-open key ranges via EXPLAIN index dives, and first/next key prefix discovery, with all key ordering done server-side under the column's own collation. Includes a LIKE-pattern escaping helper so prefixes containing wildcard characters match literally. Covered by unit tests plus a live-MySQL test (opt-in via MZ_TEST_MYSQL_URL) that exercises EXPLAIN estimates, LIKE escaping against metacharacter keys, and range bounds on both probes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add the MySql service to the cargo-test composition and hand its URL to nextest as MZ_TEST_MYSQL_URL, following the pattern POSTGRES_URL and METADATA_BACKEND_URL already use. Tests gated on the variable skip locally when it is unset but panic when CI is set, matching the timestamp oracle's tripwire so a wiring regression cannot silently retire them. Tests sharing the server run concurrently under nextest, so each must confine itself to a uniquely named scratch database it creates itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Cover ULID keys (long shared timestamp prefix), UUID keys, keys built from LIKE metacharacters, case-insensitive vs binary collations, and stale table statistics. The stale statistics test pins down that range estimates come from index dives on the real B-tree, so they stay accurate even while information_schema.tables reports 0 rows. The metacharacter test asserts the partition property of a prefix walk, since a key shorter than the prefix length subsumes longer keys sharing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A key shorter than the prefix length names an exact key, and the LIKE anchor skipped every key extending it, leaving such ranges unsplittable at any depth. next_prefix now steps just past the exact key so its extensions become prefixes of their own. Tests: pin charset and collation explicitly everywhere, share table setup through helpers, and reorganize so behavior-explaining tests lead and helpers trail. New coverage: basic and case-insensitive traversal, multibyte keys, EXPLAIN estimate sizing, and sargability asserted via session handler counters, with a self-check that a non-sargable query trips them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Move the broader live-MySQL test suite (case sensitivity, wildcards, multibyte data, ULID/UUID keys, LIKE metacharacters, collations, stale statistics, sargability) to a stacked follow-up branch so this PR stays focused on the probing interface itself.

Nothing outside the probe module uses the LIKE escaping, keeping it private leaves the NO_BACKSLASH_ESCAPES caveat an internal note rather than a public contract.

estimate_range_rows returns the optimizer estimate as an Option instead of defaulting to 0, and its doc records observed accuracy on a large static table. The next-prefix LIKE anchor declares an explicit ESCAPE so the pattern no longer depends on the sql_mode default escape character, making NO_BACKSLASH_ESCAPES sessions behave identically. Comments document the charset conversion reasoning and the connection charset assumption, and range end-bound SQL assembly is deduplicated into a helper.

Take a concrete mysql_async::Conn instead of a Queryable generic. Replace the inclusive lower bound with an exclusive one throughout, a key exactly equal to a bound is skipped as a split point and its extensions surface through the exclusive bound on re-splits. Decompose the next-prefix probe into max_key_with_prefix plus prefix_of_first_key_in_range, drop the client-side character counting entirely, and rename the probes to say what they return. Privatize explain_row_estimate, estimates are reached through KeyProber.

The anchor and seek probes read separate snapshots outside a transaction, and an insert matching the prefix between them makes the walk see the same prefix again instead of advancing.

Both prefix probes take max_prefix_length, replacing the mismatched prefix_len and len, and docs say "up to" to match the shorter-key behavior.

upper_bound becomes upper_bound_exclusive to match lower_bound_exclusive, and estimate_range_rows takes the same names. Docs state that both bounds are exclusive.

first becomes first_key_in_range and next becomes first_row_not_matching_prefix, the probe method names minus the shared prefix_of_ stem, short enough that every assertion stays a one-liner.

max_key_with_prefix splices range_filter output after its LIKE condition instead of using a bespoke append helper. The TRUE fallback makes that composition uniform, the clause stays a valid predicate after WHERE or AND even with no bounds.

The test wrappers take the full probe method names, trading one-line assertions for grep-identical naming, and the range_filter doc is condensed.

estimate_range_rows accepts an open lower bound like the prefix probes, and callers pass None for the start of the key space instead of an empty string sentinel.

query_string mapped values that fail UTF-8 decoding to None, silently ending prefix walks early. It now returns MySqlError::NonUtf8KeyValue so callers can log the condition and fall back explicitly instead of mistaking it for an exhausted range.

Restores the broader live test suite on top of the probe interface PR: case-insensitive traversal, LIKE wildcard and metacharacter data, multibyte and emoji keys, ULID and UUID primary keys, collation behavior, stale table statistics, and sargability via Handler_read session counters.

A latin1 table exercises the column-to-connection charset conversion, including the exact-key step on a key that is one character but two UTF-8 bytes. A binary key column pins the defensive behavior for invalid UTF-8: decode failures read as "no next prefix" and end the walk early instead of erroring. setup_table now derives the charset from the collation name instead of hardcoding utf8mb4.

Discovers boundaries that split a table string primary key space into per-worker ranges of roughly equal estimated row counts. Ranges the optimizer estimates too large are recursively subdivided at each distinct key prefix one character longer, probing through KeyProber, then accumulated into per-worker buckets, so discovery costs EXPLAIN index dives instead of an O(rows) index pass. Inaccurate estimates skew bucket sizes but never correctness: any ordered boundary list partitions the key space. All key ordering happens server-side under the column collation. The walk guards against non-advancing prefixes and caps children per split so a misbehaving server cannot hang it. KeyProber steps past exact keys shorter than the prefix length, so a lone short key among keys extending it cannot leave a range unsplittable. Also documents the caller contracts on like_prefix_pattern and explain_row_estimate.

Drop MAX_DEPTH, MAX_CHILDREN_PER_SPLIT, and the non-advancing prefix guard. On healthy data the walk terminates because child ranges shrink and fresh estimates track them. The pathological cases (phantom estimates, misbehaving servers) will be bounded by the per-table request budget once it lands, rather than by per-mechanism caps. Reformulate bucket sizing as a per-worker share divided by BUCKETS_PER_WORKER, dropping the double-to-8 bucket floor for small worker counts.

partition() becomes a composition of three stages: bucket_target_rows (pure sizing math), split_into_ranges (the only stage touching PartitionDb), and assign_boundaries (pure bucket accumulation). The pure stages are now directly unit-testable and the signatures document the data flow.

Rename the sizing knobs to say what they mean (TARGET_RANGES_PER_WORKER, min_rows_per_worker, target_max_rows_per_range), trim module and function docs to the essentials, and stop deduplicating repeated boundary ends. Duplicate ends only arise from non-advancing servers, and the snapshot layer validates boundary monotonicity server-side before using boundaries.

MysqlKeyProber is the concrete prober over a live connection, and the partitioner trait seam takes the KeyProber name. explain_row_estimate is no longer part of the crate API, callers reach estimates through MysqlKeyProber.

Mock-database tests for bucket assignment, empty and small tables, short exact keys among extending keys, and non-advancing prefixes, plus a live MySQL test covering EXPLAIN estimates over prepared statements, LIKE pattern semantics, and server-side validation that boundaries are strictly increasing and partition the table.

Replace the OFFSET-walking boundary discovery with the prefix-based partitioner in mz-mysql-util, so discovery costs EXPLAIN index dives instead of an O(rows) index pass. Only string primary keys are supported. Integer keys, which the OFFSET walk used to sample, now fall back to a single-worker whole-table read. Prefixes of a numeric key do not order consistently with its values, so they would need a separate numeric range splitter. Boundaries are rendered as SQL literals via the server QUOTE() and still pass the existing strict-monotonicity verification in each read transaction. The new mysql_source_snapshot_partition_min_rows dyncfg (default 50000) stops splitting below a minimum range size. Test configs set it low so the tiny tables in mysql-cdc testdrive and parallel-workload still exercise range reads.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from 57d29d8 to 777e7e8 Compare July 31, 2026 23:50

peterdukelarsen mentioned this pull request Aug 3, 2026

Set up methods for quickly probing string pk space #38022

Open

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch 2 times, most recently from 5fc40ca to fe9487a Compare August 4, 2026 00:24

peterdukelarsen and others added 9 commits August 3, 2026 20:39

mysql-util: strip comments from probe module and CI wiring

6bc5a6b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: document key probe methods

63e9abb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: move exact-key step rationale to the decision point

321083e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: make like_prefix_pattern private and tighten a comment

aab0ebf

Nothing outside the probe module uses the LIKE escaping, keeping it private leaves the NO_BACKSLASH_ESCAPES caveat an internal note rather than a public contract.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch 3 times, most recently from f5b7347 to 2094203 Compare August 4, 2026 20:57

peterdukelarsen added 2 commits August 4, 2026 17:43

mysql-util: make doc URLs hyperlinks to satisfy rustdoc lints

6180c58

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch 3 times, most recently from ccf5ce7 to d454612 Compare August 5, 2026 17:18

peterdukelarsen added 3 commits August 5, 2026 15:31

mysql-util: document the transaction requirement on the two-probe walk

77be8ab

The anchor and seek probes read separate snapshots outside a transaction, and an insert matching the prefix between them makes the walk see the same prefix again instead of advancing.

mysql-util: name prefix length arguments consistently

cd6c641

Both prefix probes take max_prefix_length, replacing the mismatched prefix_len and len, and docs say "up to" to match the shorter-key behavior.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from d454612 to 381e6c6 Compare August 5, 2026 20:16

mysql-util: name both range bounds as exclusive

74f4314

upper_bound becomes upper_bound_exclusive to match lower_bound_exclusive, and estimate_range_rows takes the same names. Docs state that both bounds are exclusive.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from 381e6c6 to da2fa1e Compare August 5, 2026 20:23

peterdukelarsen added 2 commits August 5, 2026 16:28

mysql-util: name test helpers after the probes they wrap

c017ca4

first becomes first_key_in_range and next becomes first_row_not_matching_prefix, the probe method names minus the shared prefix_of_ stem, short enough that every assertion stays a one-liner.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from da2fa1e to 92b6dee Compare August 5, 2026 20:35

mysql-util: name test wrappers after the probes they call

79df661

The test wrappers take the full probe method names, trading one-line assertions for grep-identical naming, and the range_filter doc is condensed.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from 92b6dee to 1738788 Compare August 5, 2026 20:49

peterdukelarsen added 2 commits August 5, 2026 16:53

mysql-util: take optional lower bounds everywhere

4eaaab8

estimate_range_rows accepts an open lower bound like the prefix probes, and callers pass None for the start of the key space instead of an empty string sentinel.

mysql-util: condense the transaction note and trim a redundant assertion

28fe555

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch 2 times, most recently from 8a67b4c to e6f18a6 Compare August 5, 2026 22:19

peterdukelarsen added 10 commits August 5, 2026 18:32

mysql-util: rename prober types

65de2bc

MysqlKeyProber is the concrete prober over a live connection, and the partitioner trait seam takes the KeyProber name. explain_row_estimate is no longer part of the crate API, callers reach estimates through MysqlKeyProber.

peterdukelarsen force-pushed the pl/mysql-snapshot-prefix-partition branch from e6f18a6 to 0344f3c Compare August 5, 2026 22:36

Navigation Menu

Uh oh!

storage: Fast approximate snapshot partitioning - #37994

storage: Fast approximate snapshot partitioning#37994
peterdukelarsen wants to merge 30 commits into
MaterializeInc:mainfrom
peterdukelarsen:pl/mysql-snapshot-prefix-partition

peterdukelarsen commented Jul 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

peterdukelarsen commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Downsides

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

peterdukelarsen commented Jul 31, 2026 •

edited

Loading