Set up methods for quickly probing string pk space by peterdukelarsen · Pull Request #38022 · MaterializeInc/materialize

peterdukelarsen · 2026-08-03T20:12:40Z

Motivation

Working to find a faster way to partition a mysql table by string primary key. See full PR draft here: #37994. It has some nice graphs 🎉📉

Part of: SS-97

Description

Set up a couple of utility methods for interacting quickly with MySQL.

Set up tests that will run in CI against MySQL to validate behavior.

Key queries or query fragments here include:

SELECT LEFT(pk_col, n) which will grab us up to n characters from a VARCHAR or CHAR(N) column. This is particularly useful to avoid needing to worry about ordering strings correctly in Rust.
EXPLAIN SELECT ... which will give us an estimate of the rows being queried. For simple queries along the primary key it will make this estimate by actually diving down the B tree, sampling a few pages, and then extrapolating from that.
SELECT pk_col from table where pk_col like "prefix%" order by pk_col DESC LIMIT 1 for getting the maximum key for a given prefix, which can then be used to get the minimum key of the next prefix. This is unfortunately complex, but is a more efficient workaround for executing SELECT LEFT(pk_col, n) WHERE pk_col > "prefix" and not like "prefix%" LIMIT 1 which for whatever reason does a scan of all ros under "prefix%".

Verification

I've hand-tested a lot of the SQL queries here, and posted some details about that here: https://linear.app/materializeinc/project/mysql-parallel-snapshotting-8a8a7a4f2f29/activity#project-update-15cbf9b9.
I've run the fully-fledged version of the full PR: storage: Fast approximate snapshot partitioning #37994.
This PR actually helped me catch a small bug where when a value is short, i.e. "a" it would have prevented discovery of "aa", "ab", "abc", etc...

Add a probe module exposing KeyProber over a string key column of a table: optimizer row count estimates for half-open key ranges via EXPLAIN index dives, and first/next key prefix discovery, with all key ordering done server-side under the column's own collation. Includes a LIKE-pattern escaping helper so prefixes containing wildcard characters match literally. Covered by unit tests plus a live-MySQL test (opt-in via MZ_TEST_MYSQL_URL) that exercises EXPLAIN estimates, LIKE escaping against metacharacter keys, and range bounds on both probes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add the MySql service to the cargo-test composition and hand its URL to nextest as MZ_TEST_MYSQL_URL, following the pattern POSTGRES_URL and METADATA_BACKEND_URL already use. Tests gated on the variable skip locally when it is unset but panic when CI is set, matching the timestamp oracle's tripwire so a wiring regression cannot silently retire them. Tests sharing the server run concurrently under nextest, so each must confine itself to a uniquely named scratch database it creates itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Cover ULID keys (long shared timestamp prefix), UUID keys, keys built from LIKE metacharacters, case-insensitive vs binary collations, and stale table statistics. The stale statistics test pins down that range estimates come from index dives on the real B-tree, so they stay accurate even while information_schema.tables reports 0 rows. The metacharacter test asserts the partition property of a prefix walk, since a key shorter than the prefix length subsumes longer keys sharing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A key shorter than the prefix length names an exact key, and the LIKE anchor skipped every key extending it, leaving such ranges unsplittable at any depth. next_prefix now steps just past the exact key so its extensions become prefixes of their own. Tests: pin charset and collation explicitly everywhere, share table setup through helpers, and reorganize so behavior-explaining tests lead and helpers trail. New coverage: basic and case-insensitive traversal, multibyte keys, EXPLAIN estimate sizing, and sargability asserted via session handler counters, with a self-check that a non-sargable query trips them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Move the broader live-MySQL test suite (case sensitivity, wildcards, multibyte data, ULID/UUID keys, LIKE metacharacters, collations, stale statistics, sargability) to a stacked follow-up branch so this PR stays focused on the probing interface itself.

Nothing outside the probe module uses the LIKE escaping, keeping it private leaves the NO_BACKSLASH_ESCAPES caveat an internal note rather than a public contract.

ublubu

I haven't looked over the test cases in detail yet. The SQL generation looks legit to my human eyeballs.

ublubu · 2026-08-04T18:26:38Z

+    /// Estimates the row count for the given range. These estimates can vary pretty widely. They
+    /// will generally never be more than half the size of the full row count reported by
+    /// `information_schema.tables`. In some tests these have been over-estimates in practice,
+    /// where the sum of all table ranges has been ~2x as large as the estimate or table size.


Suggested change

/// Estimates the row count for the given range. These estimates can vary pretty widely. They

/// will generally never be more than half the size of the full row count reported by

/// `information_schema.tables`. In some tests these have been over-estimates in practice,

/// where the sum of all table ranges has been ~2x as large as the estimate or table size.

/// Estimates the row count for the given range. Estimates vary widely

/// but are generally < half the `TABLE_ROWS` reported by `information_schema.tables`.

/// In some tests, the sum of all table ranges (actual or estimated by this query?

/// has been ~2x the estimate (Which estimate is this?)

/// or table size (Actual table size or the `TABLE_ROWS` statistic?).

generally never
In some tests ... in practice

Updated to something like this:

/// Estimates the row count for the given range. Estimates vary widely. On a static table /// with 2.2B rows we observed estimates that should be near 2B report exactly half the /// `TABLE_ROWS` reported by `information_schema.tables`. The sum of the row estimates /// from this function were around 4B for the same test case, or about a 2x overcount relative /// to the 2.05B reported by `TABLE_ROWS` from `information_schema.tables` and the 2.2B /// actual rows. The underlying estimates are computed by sampling a small number of pages /// after traversing the index (assuming this is a primary key being filtered on), so extrapolated /// row counts can be innaccurate but appear to eventually converge towards more accurate estimates /// as the sampled range shrinks on a static table.

Stuck more strictly to observed behavior and clarified the source of the information i'm comparing.

ublubu · 2026-08-04T18:32:08Z

+        );
+        let estimate =
+            explain_row_estimate(&mut *self.conn, &select, Params::Positional(params)).await?;
+        Ok(estimate.unwrap_or(0))


Does 0 ever actually mean zero? Do we want to keep the Option wrapper here?

Good call, I've moved it to an Option

ublubu · 2026-08-04T18:41:51Z

+    /// Grabs a prefix of length `len` for the first row in the given range. If the string is
+    /// shorter than `len`, it will return that shorter value.
+    ///
+    /// The query will generally look something like:


Re: generally

Does it ever not look something like this?

sometimes the upper bound AND pk_col < 'ac' will be missing if it's an empty optional, otherwise this is pretty much the shape

ublubu · 2026-08-04T18:55:19Z

+    /// SELECT LEFT(pk_col, 3) FROM table
+    /// WHERE pk_col > (
+    ///     SELECT pk_col FROM table
+    ///     WHERE pk_col LIKE 'abc%' AND pk_col < 'ac'


Took me a minute to understand that 'ac' is end in the example.

Not sure why, given that I had no such trouble with the doc comment for first_prefix above.

Updated to this for hopefully more clarity, lmk if you think it makes it worse

/// /// ```sql /// SELECT LEFT(pk_col, /* len */ 3) FROM table /// WHERE pk_col > ( /// SELECT pk_col FROM table /// WHERE pk_col LIKE /* cur% */ 'abc%' AND pk_col < /* end */ 'ac' /// ORDER BY pk_col DESC /// LIMIT 1 /// ) AND pk_col < /* end */ 'ac' /// ORDER BY pk_col /// LIMIT 1 /// ```

ublubu · 2026-08-04T18:58:01Z

+        len: usize,
+    ) -> Result<Option<String>, MySqlError> {
+        // chars().count() counts Unicode code points, the same unit LEFT and
+        // CHAR_LENGTH use for utf8mb4 data, so the short-key check agrees


Are we always dealing with utf8mb4 data?

In rust we're dealing with UTF8 up to 4byte chars, like utf8mb4 in MySQL.

MySQL can vary, but we're relying on it serving us string-typed data as the UTF8-encoded version. So if you had something more random like latin1 it should convert to the equivalent code-point in UTF8.

I haven't been able to find counter-examples where this doesn't work and I'll have some tests (and already have some integration tests for varied charsets from previous iterations). Open to falling back to selecting out CHAR_LENGTH directly instead if this feels too risky -- wasn't able to find a clean page of docs confirming this behavior.

For now I went ahead and clarified the comment a bit to make the assumption clear. LMK if you have thoughts about testing this vs. taking a different approach. Two lines of defense we have against anything too bad happening are:

Forcing the comparison in the column's collation before using the results

Limiting the number of calls we make to MySQL for this probing which would help with any unfortunate infinite loops

ublubu · 2026-08-04T18:58:34Z

+        // with how the prefix was produced.
+        if cur.chars().count() < len {
+            // When `cur` has fewer than `len` characters it names an exact key
+            // rather than a truncation, and the anchor would skip every key extending


What is anchor? This feels like Claude inventing its own jargon again.

This section took some digging to understand. The comment misstates the intent of the logic.

I think this is a better representation of what we're doing:

When we ask for next_prefix(cur='az', len=2), the next row after prefix 'az' might be the too-short pk_col='b'. That gives us cur='b'.

Our goal is to partition the key space into a sequence of prefixes of len=2.
For example, next_prefix(cur='b', len=2) should be 'ba', not 'ca'.

If we use pk_col LIKE 'b%', we'd match all prefixes 'ba', 'bb', ... into the same partition.
Therefore, we define the cur='b' partition with the filter pk_col = 'b',
and the first row of the next partition is pk_col > 'b' LIMIT 1.

What is anchor? This feels like Claude inventing its own jargon again.

Yeah... anchor is just the prefix represented by cur.

I think this is a better representation of what we're doing: ...

I don't think this matches my intent. What I'm trying to cover is actually the case like:

pks: ["a", "aa, "aaa", "bbb"]

If my current prefix is "a" and my target depth is 2 and my upper bound is "b" from the previous walk. If I fill out my query I get:

SELECT LEFT(pk_col, /* len */ 2) FROM table WHERE pk_col > ( SELECT pk_col FROM table WHERE pk_col LIKE /* cur% */ 'a%' AND pk_col < /* end */ 'b' ORDER BY pk_col DESC LIMIT 1 ) AND pk_col < /* end */ 'b' ORDER BY pk_col LIMIT 1

This will be empty. The inner clause will resolve to "aaa" because we're finding the last key in the a.* range.

Updated comment to:

// When `cur` has fewer than `len` characters it likely names an exact key, i.e. imagine strings // "a", "aa", "aaa". If we get "a" with depth 2 we want the next key to be "aa". Without this block, // we'd skip past all of these strings because they match the prefix "a".

ublubu · 2026-08-04T19:37:12Z

+                col = self.col,
+                table = self.table,
+            );
+            params.insert(0, u64::cast_from(len).into());


It's kinda weird seeing this happen after self.range_filter above. So this inserts a param and shifts everything else to the right?

Yeah, I think it makes sense if you notice that the ? for the SQL here is before the {clause} we got back from range_filter, so since the params are positional we need the new param to go first.

ublubu · 2026-08-04T19:42:44Z

+        if let Some(end) = end {
+            sql.push_str(&format!(" AND {col} < ?"));
+            params.push(end.into());
+        }


This feels like its own helper function, especially since it also exists in range_filter.

I like the idea of having a fn less_than_end(&mut sql, &mut params, end: Option<_>) separate from
fn col_cmp_start(&mut sql, &mut params, start, cmp: &str)

ublubu · 2026-08-04T19:43:41Z

+fn like_prefix_pattern(prefix: &str) -> String {
+    let mut pattern = String::with_capacity(prefix.len() + 1);
+    for c in prefix.chars() {
+        if matches!(c, '\\' | '%' | '_') {


Are we absolutely sure these the only special characters? Might be good to link those docs here.

LIKE allows special characters '%' and '_' and then backslash is the escaping character. Docs here for LIKE: https://dev.mysql.com/doc/refman/8.4/en/pattern-matching.html.

To simplify and make this more resilient I'm actually going to use an explicit escape character -- this could have been a little broken/messed up if someone had disabled backslash escaping.

estimate_range_rows returns the optimizer estimate as an Option instead of defaulting to 0, and its doc records observed accuracy on a large static table. The next-prefix LIKE anchor declares an explicit ESCAPE so the pattern no longer depends on the sql_mode default escape character, making NO_BACKSLASH_ESCAPES sessions behave identically. Comments document the charset conversion reasoning and the connection charset assumption, and range end-bound SQL assembly is deduplicated into a helper.

peterdukelarsen

Thanks for the review!

peterdukelarsen · 2026-08-04T20:09:44Z

+    /// Estimates the row count for the given range. These estimates can vary pretty widely. They
+    /// will generally never be more than half the size of the full row count reported by
+    /// `information_schema.tables`. In some tests these have been over-estimates in practice,
+    /// where the sum of all table ranges has been ~2x as large as the estimate or table size.


Updated to something like this:

/// Estimates the row count for the given range. Estimates vary widely. On a static table /// with 2.2B rows we observed estimates that should be near 2B report exactly half the /// `TABLE_ROWS` reported by `information_schema.tables`. The sum of the row estimates /// from this function were around 4B for the same test case, or about a 2x overcount relative /// to the 2.05B reported by `TABLE_ROWS` from `information_schema.tables` and the 2.2B /// actual rows. The underlying estimates are computed by sampling a small number of pages /// after traversing the index (assuming this is a primary key being filtered on), so extrapolated /// row counts can be innaccurate but appear to eventually converge towards more accurate estimates /// as the sampled range shrinks on a static table.

Stuck more strictly to observed behavior and clarified the source of the information i'm comparing.

peterdukelarsen · 2026-08-04T20:11:52Z

+        );
+        let estimate =
+            explain_row_estimate(&mut *self.conn, &select, Params::Positional(params)).await?;
+        Ok(estimate.unwrap_or(0))


Good call, I've moved it to an Option

peterdukelarsen · 2026-08-04T20:14:48Z

+    /// Grabs a prefix of length `len` for the first row in the given range. If the string is
+    /// shorter than `len`, it will return that shorter value.
+    ///
+    /// The query will generally look something like:


sometimes the upper bound AND pk_col < 'ac' will be missing if it's an empty optional, otherwise this is pretty much the shape

peterdukelarsen · 2026-08-04T20:20:48Z

+    /// SELECT LEFT(pk_col, 3) FROM table
+    /// WHERE pk_col > (
+    ///     SELECT pk_col FROM table
+    ///     WHERE pk_col LIKE 'abc%' AND pk_col < 'ac'


Updated to this for hopefully more clarity, lmk if you think it makes it worse

/// /// ```sql /// SELECT LEFT(pk_col, /* len */ 3) FROM table /// WHERE pk_col > ( /// SELECT pk_col FROM table /// WHERE pk_col LIKE /* cur% */ 'abc%' AND pk_col < /* end */ 'ac' /// ORDER BY pk_col DESC /// LIMIT 1 /// ) AND pk_col < /* end */ 'ac' /// ORDER BY pk_col /// LIMIT 1 /// ```

peterdukelarsen · 2026-08-04T20:53:18Z

+        len: usize,
+    ) -> Result<Option<String>, MySqlError> {
+        // chars().count() counts Unicode code points, the same unit LEFT and
+        // CHAR_LENGTH use for utf8mb4 data, so the short-key check agrees


In rust we're dealing with UTF8 up to 4byte chars, like utf8mb4 in MySQL.

MySQL can vary, but we're relying on it serving us string-typed data as the UTF8-encoded version. So if you had something more random like latin1 it should convert to the equivalent code-point in UTF8.

I haven't been able to find counter-examples where this doesn't work and I'll have some tests (and already have some integration tests for varied charsets from previous iterations). Open to falling back to selecting out CHAR_LENGTH directly instead if this feels too risky -- wasn't able to find a clean page of docs confirming this behavior.

peterdukelarsen · 2026-08-04T21:01:32Z

+        // with how the prefix was produced.
+        if cur.chars().count() < len {
+            // When `cur` has fewer than `len` characters it names an exact key
+            // rather than a truncation, and the anchor would skip every key extending


What is anchor? This feels like Claude inventing its own jargon again.

Yeah... anchor is just the prefix represented by cur.

I think this is a better representation of what we're doing: ...

I don't think this matches my intent. What I'm trying to cover is actually the case like:

pks: ["a", "aa, "aaa", "bbb"]

If my current prefix is "a" and my target depth is 2 and my upper bound is "b" from the previous walk. If I fill out my query I get:

SELECT LEFT(pk_col, /* len */ 2) FROM table WHERE pk_col > ( SELECT pk_col FROM table WHERE pk_col LIKE /* cur% */ 'a%' AND pk_col < /* end */ 'b' ORDER BY pk_col DESC LIMIT 1 ) AND pk_col < /* end */ 'b' ORDER BY pk_col LIMIT 1

This will be empty. The inner clause will resolve to "aaa" because we're finding the last key in the a.* range.

peterdukelarsen · 2026-08-04T21:03:56Z

+                col = self.col,
+                table = self.table,
+            );
+            params.insert(0, u64::cast_from(len).into());


Yeah, I think it makes sense if you notice that the ? for the SQL here is before the {clause} we got back from range_filter, so since the params are positional we need the new param to go first.

peterdukelarsen · 2026-08-04T21:14:37Z

+        len: usize,
+    ) -> Result<Option<String>, MySqlError> {
+        // chars().count() counts Unicode code points, the same unit LEFT and
+        // CHAR_LENGTH use for utf8mb4 data, so the short-key check agrees


For now I went ahead and clarified the comment a bit to make the assumption clear. LMK if you have thoughts about testing this vs. taking a different approach. Two lines of defense we have against anything too bad happening are:

Forcing the comparison in the column's collation before using the results

Limiting the number of calls we make to MySQL for this probing which would help with any unfortunate infinite loops

peterdukelarsen · 2026-08-04T21:17:58Z

+        // with how the prefix was produced.
+        if cur.chars().count() < len {
+            // When `cur` has fewer than `len` characters it names an exact key
+            // rather than a truncation, and the anchor would skip every key extending


Updated comment to:

// When `cur` has fewer than `len` characters it likely names an exact key, i.e. imagine strings // "a", "aa", "aaa". If we get "a" with depth 2 we want the next key to be "aa". Without this block, // we'd skip past all of these strings because they match the prefix "a".

peterdukelarsen · 2026-08-04T21:41:34Z

+fn like_prefix_pattern(prefix: &str) -> String {
+    let mut pattern = String::with_capacity(prefix.len() + 1);
+    for c in prefix.chars() {
+        if matches!(c, '\\' | '%' | '_') {


LIKE allows special characters '%' and '_' and then backslash is the escaping character. Docs here for LIKE: https://dev.mysql.com/doc/refman/8.4/en/pattern-matching.html.

To simplify and make this more resilient I'm actually going to use an explicit escape character -- this could have been a little broken/messed up if someone had disabled backslash escaping.

martykulma

Thanks @peterdukelarsen, looking good!

Per our conversation:
Looking at this PR in isolation is a little challenging without knowing the algorithm. I'd love to have a clearer idea of what the overall algorithm is to make sense of some of these methods. It might make sense to rename the methods to carry more semantic meaning for that algorithm (e.g. max_key_with_prefix) as well.

I strongly recommend avoiding the situation where start can be inclusive or exclusive (e.g. col >= a / col > a). It would be easier on future us to not have to reason about when it should be inclusive vs. exclusive and just reason about what the start/end are.

martykulma · 2026-08-05T16:50:19Z

+            let (clause, mut params) = self.range_filter(cur, false, end);
+            let sql = format!(
+                "SELECT LEFT({col}, ?) FROM {table} WHERE {clause} ORDER BY {col} LIMIT 1",
+                col = self.col,
+                table = self.table,
+            );


This looks very similar to the above first_prefix, with the difference between the two being start inclusivity. Can we find a way to merge these queries? The conditional inclusivity is also a little concerning. It's easy to get that boolean wrong, so worth considering a solution doesn't rely on it.

peterdukelarsen · 2026-08-05T18:01:14Z

Thanks for the review Marty!

At a high level the intended algorithm is this (copied from #38047):

Grab all of the unique first characters of the primary key strings and use EXPLAIN to estimate their row count
For any character with a high row count we will redo the process one character deeper
We continue stepping down into longer prefixes until we run out of a configured number of calls to make or we've resolved granular enough buckets to build partition boundaries
Finally, we pick partition boundaries based on the total estimated row count (which could diverge from the real row count or the regular estimated row count by a good bit), and just walk the partitions in order computing boundary keys as evenly spaced as possible.

The specific queries and the implementation use ranges more heavily to handle an edge case and an optimization:

Edge case: keys shorter than the prefix throw a wrench in representing ranges as prefixes. i.e. imagine keys "a", "aa", "aaa". If we represented a prefix as "aa", then "a" wouldn't be included in "aa%".
Optimization: once we get an estimate under our target estimated row count for the rest of a prefix we would like to exit early.

A current full draft of the algorithm is available in https://github.com/MaterializeInc/materialize/pull/38047/changes at src/mysql-util/src/partition.rs.

Take a concrete mysql_async::Conn instead of a Queryable generic. Replace the inclusive lower bound with an exclusive one throughout, a key exactly equal to a bound is skipped as a split point and its extensions surface through the exclusive bound on re-splits. Decompose the next-prefix probe into max_key_with_prefix plus prefix_of_first_key_in_range, drop the client-side character counting entirely, and rename the probes to say what they return. Privatize explain_row_estimate, estimates are reached through KeyProber.

The anchor and seek probes read separate snapshots outside a transaction, and an insert matching the prefix between them makes the walk see the same prefix again instead of advancing.

peterdukelarsen · 2026-08-05T19:46:18Z

Ok, I've killed the different >=/<= clauses. I'm now planning to handle the case of short keys differently now, where we'll simply skip over short primary keys if depth is >= to them.

Consider these primary keys:

a
aa
ab
ac
b
ba
bb
bc

At depth 1 we'll have recorded two ranges: [a, b), [b, None)

At depth 2 before we would have computed something like:
[a, aa), [aa, ab), [ab, ac), [ac, b)
[b, ba), [ba, bb), [bb, bc), [bc, None)

Now it will just be:
[aa, ab), [ab, ac), [ac, b)
[ba, bb), [bb, bc), [bc, None)

You can imagine sort of jagged degenerate cases like this:

a
aa
aaa
aaaa

These were already degenerate though. In general we're dropping one row on the floor at each depth and we're not planning to compute with more granularity than maybe 10k rows per prefix, so losing 1 row at a given prefix should be acceptable.

Both prefix probes take max_prefix_length, replacing the mismatched prefix_len and len, and docs say "up to" to match the shorter-key behavior.

upper_bound becomes upper_bound_exclusive to match lower_bound_exclusive, and estimate_range_rows takes the same names. Docs state that both bounds are exclusive.

first becomes first_key_in_range and next becomes first_row_not_matching_prefix, the probe method names minus the shared prefix_of_ stem, short enough that every assertion stays a one-liner.

max_key_with_prefix splices range_filter output after its LIKE condition instead of using a bespoke append helper. The TRUE fallback makes that composition uniform, the clause stays a valid predicate after WHERE or AND even with no bounds.

The test wrappers take the full probe method names, trading one-line assertions for grep-identical naming, and the range_filter doc is condensed.

estimate_range_rows accepts an open lower bound like the prefix probes, and callers pass None for the start of the key space instead of an empty string sentinel.

query_string mapped values that fail UTF-8 decoding to None, silently ending prefix walks early. It now returns MySqlError::NonUtf8KeyValue so callers can log the condition and fall back explicitly instead of mistaking it for an exhausted range.

martykulma

Nice - nothing blocking from my perspective!

🐟

martykulma · 2026-08-06T16:26:39Z

+            "SELECT {col} FROM {table} WHERE {col} LIKE ? ESCAPE '{LIKE_ESCAPE}' \
+             AND {range_clause} ORDER BY {col} DESC LIMIT 1",


Not blocking, but this might still cause an issue. My understanding is that for the first iteration of this, range_clause is TRUE as, is that correct?

I think this should be fine. If we called this with the root prefix of None this would return the max row in the database. If we tried to find the row after the max row or the row past the root prefix we would (correctly) get nothing.

So, to step one character deeper we call prefix_of_first_key_in_range i.e. None might spit out "a" and "a" might spit out "aa".

Then to step through siblings at the same level we call: prefix_of_first_row_not_matching_prefix

From a correctness perspective, it should be fine. Concerned the query plan might be worse than expected without range_clause=TRUE. Out of curiosity, have you run explain on the query with root prefix of None on the reproduction table? I didn't see it in the linear notes (might have missed it).

martykulma · 2026-08-06T16:32:37Z

+                qualified_table_name: self.table.clone(),
+                column_name: self.col.clone(),


It looks like this doesn't match the other MySqlError variants, which report the unquoted name as qualified_table_name.

You may also want to record the actual error instead of discarding (e.g. Err(_))

martykulma · 2026-08-06T16:52:44Z

+    }
+
+    /// Returns the prefix of up to `max_prefix_length` characters of the first key after `prefix`,
+    /// but below `upper_bound_exclusive`. Returns None if no key matching these conditions exists.


My understanding is that this returns None if there are no matches for the prefix OR there are no keys after that are below the upper_bound_exclusive.

Does this then require that callers run prefix_of_first_key_in_range first?

Yeah, that's right and to use it as intended we'll need to call prefix_of_first_key_in_range.

Basically, prefix_of_first_key_in_range let's you go one character deeper, relying on the fact that the prefix will be less than all keys in it's owned range except for a short key matching the prefix exactly.

…alue The variant matches the other MySqlError variants: qualified_table_name carries the unquoted schema.table, and the underlying decode error is recorded instead of discarded.

peterdukelarsen marked this pull request as ready for review August 3, 2026 23:11

peterdukelarsen requested review from a team as code owners August 3, 2026 23:12

peterdukelarsen mentioned this pull request Aug 3, 2026

Flesh out testing and fix bugs for string primary key probes #38030

Open

peterdukelarsen and others added 9 commits August 3, 2026 20:39

mysql-util: strip comments from probe module and CI wiring

6bc5a6b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: document key probe methods

63e9abb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: move exact-key step rationale to the decision point

321083e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mysql-util: make like_prefix_pattern private and tighten a comment

aab0ebf

Nothing outside the probe module uses the LIKE escaping, keeping it private leaves the NO_BACKSLASH_ESCAPES caveat an internal note rather than a public contract.

peterdukelarsen force-pushed the pl/mysql-key-probes branch from b407472 to aab0ebf Compare August 4, 2026 00:41

peterdukelarsen changed the title ~~storage: Set up methods for quickly probing string pk space~~ Set up methods for quickly probing string pk space Aug 4, 2026

ublubu reviewed Aug 4, 2026

View reviewed changes

peterdukelarsen commented Aug 4, 2026

View reviewed changes

mysql-util: make doc URLs hyperlinks to satisfy rustdoc lints

6180c58

peterdukelarsen requested review from a team and ublubu August 4, 2026 22:01

martykulma reviewed Aug 5, 2026

View reviewed changes

peterdukelarsen added 2 commits August 5, 2026 15:31

mysql-util: document the transaction requirement on the two-probe walk

77be8ab

The anchor and seek probes read separate snapshots outside a transaction, and an insert matching the prefix between them makes the walk see the same prefix again instead of advancing.

peterdukelarsen added 4 commits August 5, 2026 16:03

mysql-util: name prefix length arguments consistently

cd6c641

Both prefix probes take max_prefix_length, replacing the mismatched prefix_len and len, and docs say "up to" to match the shorter-key behavior.

mysql-util: name both range bounds as exclusive

74f4314

upper_bound becomes upper_bound_exclusive to match lower_bound_exclusive, and estimate_range_rows takes the same names. Docs state that both bounds are exclusive.

mysql-util: name test helpers after the probes they wrap

c017ca4

first becomes first_key_in_range and next becomes first_row_not_matching_prefix, the probe method names minus the shared prefix_of_ stem, short enough that every assertion stays a one-liner.

peterdukelarsen added 3 commits August 5, 2026 16:45

mysql-util: name test wrappers after the probes they call

79df661

The test wrappers take the full probe method names, trading one-line assertions for grep-identical naming, and the range_filter doc is condensed.

mysql-util: take optional lower bounds everywhere

4eaaab8

estimate_range_rows accepts an open lower bound like the prefix probes, and callers pass None for the start of the key space instead of an empty string sentinel.

mysql-util: condense the transaction note and trim a redundant assertion

28fe555

peterdukelarsen requested a review from martykulma August 5, 2026 22:07

peterdukelarsen force-pushed the pl/mysql-key-probes branch 2 times, most recently from b8c4bd1 to c30677d Compare August 6, 2026 14:08

peterdukelarsen force-pushed the pl/mysql-key-probes branch from c30677d to 96b5e61 Compare August 6, 2026 14:09

martykulma approved these changes Aug 6, 2026

View reviewed changes

mysql-util: report unquoted names and the decode error in NonUtf8KeyV…

56defea

…alue The variant matches the other MySqlError variants: qualified_table_name carries the unquoted schema.table, and the underlying decode error is recorded instead of discarded.

peterdukelarsen enabled auto-merge (squash) August 6, 2026 17:50

peterdukelarsen merged commit 2c7b27f into MaterializeInc:main Aug 6, 2026
129 of 130 checks passed

peterdukelarsen mentioned this pull request Aug 6, 2026

Flesh out testing for string primary key probes #38092

Draft

-    /// Estimates the row count for the given range. These estimates can vary pretty widely. They
-    /// will generally never be more than half the size of the full row count reported by
-    /// `information_schema.tables`. In some tests these have been over-estimates in practice,
-    /// where the sum of all table ranges has been ~2x as large as the estimate or table size.
+    /// Estimates the row count for the given range. Estimates vary widely
+    /// but are generally < half the `TABLE_ROWS` reported by `information_schema.tables`.
+    /// In some tests, the sum of all table ranges (actual or estimated by this query?
+    /// has been ~2x the estimate (Which estimate is this?)
+    /// or table size (Actual table size or the `TABLE_ROWS` statistic?).

		"SELECT {col} FROM {table} WHERE {col} LIKE ? ESCAPE '{LIKE_ESCAPE}' \
		AND {range_clause} ORDER BY {col} DESC LIMIT 1",

		qualified_table_name: self.table.clone(),
		column_name: self.col.clone(),

Uh oh!

Conversation

peterdukelarsen commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

ublubu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen left a comment

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen commented Aug 3, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 4, 2026 •

edited

Loading

peterdukelarsen Aug 6, 2026 •

edited

Loading