MDEV-40168 wip - #5620
Conversation
|
|
8a9c029 to
21cf57e
Compare
21cf57e to
d3452c7
Compare
| t1 CREATE TABLE `t1` ( | ||
| `c` int(11) DEFAULT NULL, | ||
| `j` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid(`j`)) | ||
| ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci |
There was a problem hiding this comment.
So, the index is now shown in SHOW CREATE TABLE ?
There was a problem hiding this comment.
rather is NOT shown in SHOW CREATE TABLE
d3452c7 to
7cfaf2b
Compare
344ad21 to
08d0e46
Compare
|
For the record: index is not visible in information_schema: create table t25 (
js json,
key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
select * from information_schema.statistics where table_name='t25'gives nothing. |
|
And this crashes: create table t25 (
js json,
key idx ((CAST(json_extract(js, '$.tags') AS CHAR(6) ARRAY)))
)engine=innodb;
insert into t25 values ('{}'); |
Isn't this the same problem as your other comment #5620 (comment)? |
TODOs on top of those in the patch diff: - EXPLAIN output should not say fulltext - check type match to avoid false negative / positive bugs in mysql - transcode the value into the index charset in mvi_encode_key
6ff4ac9 to
78f3529
Compare
Fixed now |
e7ff23f to
d14f631
Compare
d14f631 to
2b86847
Compare
Add comments
Factor out common code into get_mvi_index()
Factor out common code into Item_func_json_contains::get_mvi_access()
Item_func_json_contains::mvi_analyze() and ::create_ft_for_mvi() were
near-identical: both checked the arguments, looked up the matching MVI,
parsed the constant second argument and ran the same scan loop calling
encode_mvi_key(). They differed only in what they did with each encoded
key.
Move all of that into get_mvi_access(), which returns an Mvi_access, and
give Mvi_access two methods:
- add_key(), to collect one encoded element key,
- create_ft_item(), to build the
MATCH vcol AGAINST ('+encoded_foo +encoded_bar ...' IN BOOLEAN MODE)
item. It honors Mvi_access::conjunctive, so JSON_OVERLAPS will get the
OR form for free.
mvi_analyze() and create_ft_for_mvi() are now thin wrappers around
get_mvi_access().
This also fixes a memory leak: the encoded keys were copied with
String::copy(), giving each String in Mvi_access::encoded a heap buffer
that is never freed (the Strings live on the MEM_ROOT, so their
destructors never run). Copy the keys onto the MEM_ROOT instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make the MVI scan a real access method: QUICK_MVI_SELECT
JSON_CONTAINS() over a multi-valued index used to be optimized by
rewriting the WHERE clause: setup_mvi_for_join() injected a synthetic
MATCH vcol AGAINST ('+k1 +k2' IN BOOLEAN MODE)
into join->conds and into select_lex->ftfunc_list, and the normal
fulltext machinery then picked it up as JT_FT access. The injected item
showed up in the plan and in the condition even though the user never
wrote it, and because it became ordinary ref access the scan was never
costed against the alternatives - it won by being in the WHERE clause.
Introduce QUICK_MVI_SELECT (QS_TYPE_MVI), a QUICK_SELECT_I that drives
the fulltext index directly through the handler API. Unlike FT_SELECT
there is no Item_func_match to have created the FT_INFO, so the quick
select creates it in reset() with ft_init_ext() and frees it with
close_search() in its destructor. Mvi_access::create_ft_item() is
replaced by build_ft_query(), which builds just the query string.
The analysis in setup_mvi_quick() is now kept: Mvi_context moves to the
header, is allocated on the mem_root and stored as JOIN::mvi_ctx, where
JOIN::get_mvi_access_for_table() looks it up. get_quick_record_count()
builds the quick select before test_quick_select() and keeps whichever
of the two is cheaper; test_quick_select() itself is untouched, so the
MVI quick is held in a local across the call (it deletes select->quick
on entry). The same save/compare is done around the second
test_quick_select() call in make_join_select(), which a LIMIT can reach.
A fulltext key never gets a bit in const_keys or keys, so mark the MVI
key of every table that has an access: the const_keys bit is what lets
the range analysis run for that table at all, the keys bit puts the
index into EXPLAIN's possible_keys.
Collect the accesses from the top-level AND-parts of the WHERE clause
only, instead of walking the whole condition. An MVI scan reads just the
rows the index matches, so it is only valid for a predicate that must
hold for every row of the result: for
json_contains(j1->'$.tags','"a"') OR json_contains(j2->'$.tags','"a"')
scanning either index would drop the rows that only match the other
branch. The deleted add_ft_for_mvi() refused COND_OR_FUNC for the same
reason; walking the condition tree lost that, which only became visible
once the accesses were actually used.
Costs are placeholders (records=10, read_time=0.001) until the engine
can estimate a fulltext search. Note that while there is no estimate,
an MVI access is also taken when test_quick_select() produced no quick
select at all, without comparing it to the cost of a table scan.
TODO: This doesn't handle UPDATE/DELETE! Should it be put into
check_quick() call?
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make optimizer trace print "range", not "index_merge" for MVI quick selects.
Add optimizer trace for the multi-valued index access
get_best_mvi_access() picked an Mvi_access and wrapped it in a
QUICK_MVI_SELECT without recording anything, so there was no way to see
which index was chosen or what it would search that index for: the
rows_estimation trace showed a range_analysis that found nothing, and
then a plan using a key the trace never mentioned.
Print a "multi_value_index_use" object:
{
"table": "t1",
"index": "idx",
"ranges": ["616161"]
}
Mvi_access::print_json() fills in the index and the element keys,
following TRP_RANGE::trace_basic_info(): same "index" / "ranges" member
names, so an MVI entry reads like a range scan's.
The keys are printed in their encoded form, which is not readable. That
is what is stored in the index and what we search for, so it is still
the useful thing to print; making it readable can come later. It is
plain ASCII (hex plus the xx/xxxx padding from encode_mvi_key()), so it
needs no JSON escaping.
get_best_mvi_access() runs inside the "rows_estimation" array, so the
named object needs an object of its own around it, the same way
make_join_statistics() and the sel_arg_alloc_limit_hit trace do it.
Without it the writer hits an assertion in
Single_line_formatting_helper::on_add_member().
The new test is a separate file because optimizer trace tests need
not_embedded.inc, and putting that in multi_valued_index.test would skip
the whole feature test on embedded builds. It cross-checks the printed
keys against mvi_encode() over the indexed column, which produces the
tokens the index is actually built from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make JSON_OVERLAPS sargable for multi-valued indexes
Both argument orders are handled, since JSON_OVERLAPS is symmetric:
JSON_OVERLAPS(array_indexed_expr, '[foo, bar, ...]')
JSON_OVERLAPS('[foo, bar, ...]', array_indexed_expr)
JSON_CONTAINS is true when ALL of the elements have a match, JSON_OVERLAPS
when ANY of them does, so the access it produces has conjunctive=false and
build_ft_query() leaves the keys optional instead of prefixing them with
'+'.
The two get_mvi_access() implementations share collect_mvi_keys(), which
is the scan of the JSON literal that used to sit inside
Item_func_json_contains::get_mvi_access().
The two differ in one way beyond the flag. An element that cannot be
encoded for the index (a number against a CHAR array, say) is skipped for
JSON_CONTAINS: dropping a key from an AND makes the index scan less
selective, so it still returns a superset of the rows the predicate
matches and the predicate does the exact filtering afterwards.
That reasoning does not hold for an OR. A row can satisfy the predicate
through the very element we failed to encode, and MVI_ENCODE skips such
elements as well, so that row has no key in the index for the scan to find
it by - dropping the key would lose it. So for a disjunctive access we
give up instead of skipping. With a row {"tags": [123]} in the table,
select * from t1 where json_overlaps(j->'$.tags','[123,"aaa"]')
must not use the index, and the test checks its result against the same
query with IGNORE INDEX.
Also print "match": "all"/"any" in the optimizer trace. Now that an access
can be either, the printed ranges alone did not say whether a row has to
have all of the keys or just one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the JSON function MVI code into opt_mvi_jsonfuncs.cc
opt_multi_valued_index.cc held two separate concerns: the index side
(how a value is encoded into the index, the access descriptor, the quick
select) and the predicate side (which JSON functions can be computed
from an MVI, which of their arguments holds the indexed expression, and
what to search the index for). Split the second one out.
Moved verbatim:
get_mvi_index()
collect_mvi_keys()
Item_func_json_contains::get_mvi_access() and ::mvi_analyze()
Item_func_json_overlaps::get_mvi_access() and ::mvi_analyze()
add_mvi_access()
The only code change is that encode_mvi_key() is no longer static: it is
used both by Item_func_mvi_encode::val_str_ascii(), which stays, and by
collect_mvi_keys(), which moves. It is declared in
opt_multi_valued_index.h now. The other three moved helpers had no
callers outside the moved code and stay static.
Item_func_mvi_encode is not a JSON function and stays put: it is how the
values get into the index in the first place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MDEV-40168: JSON-over-fulltext: add estimates.
Add records_in_range-like estimates for fulltext index
Estimate the number of records an MVI access will read
QUICK_MVI_SELECT carried records=10 and read_time=0.001, numbers picked
low enough that the access always won over a table scan. Ask the engine
instead, through the fulltext_estimate() added by the previous commit.
Mvi_access::estimate_records() estimates one element key at a time and
combines the answers the way the query combines the keys:
- A disjunctive access (JSON_OVERLAPS) reads the rows of every key, so
the estimates add up.
- A conjunctive access (JSON_CONTAINS) reads the rows that have all of
the keys, so the rarest key alone bounds the result. We use its
estimate and drop the other keys from the query: reading the rarest
key and letting the WHERE clause discard the rest is not worse than
having the engine intersect the terms. This is the trade-off
collect_mvi_keys() already makes for the keys it cannot encode - a
shorter AND matches a superset of the rows, and the JSON predicate
does the exact filtering.
The engine may be unable to estimate a key: ha_innobase only looks at
the fulltext auxiliary tables, so until the words are flushed out of the
FTS cache the answer is "unknown" for everything. Such a key takes no
part in the choice of the rarest one, and if not a single key could be
estimated the old guess stands and the query is left as it is.
For an OR we cannot do that: we have to read that key and have no idea
what it costs. Give the access a DBL_MAX read_time and do not use it.
That has to be acted on in get_best_mvi_access() rather than left to the
cost comparison, because best_access_path() takes a quick select to be
cheaper than a table scan without checking - true of anything the range
optimizer proposes, but this access does not come from there.
read_time is now the cost of reading the estimated rows plus evaluating
the WHERE clause on them, which is what the join optimizer expects of a
quick select's read_time. It does not account for the fulltext search
that produces the rowids in the first place.
The trace prints the estimate, or says the access is unusable and why.
The two existing tests ran on tables whose words were still in the FTS
cache, so the JSON_OVERLAPS sections would have stopped using the index
entirely. They now flush the cache with OPTIMIZE TABLE under
innodb_optimize_fulltext_only, the way innodb_fts.estimate does. The
trace test gets a table with a skewed distribution ("bbb" in every row,
"aaa" in one) to show the conjunctive access keeping only the rarest key
and still producing the same rows as the same query with IGNORE INDEX.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MDEV-40168: JSON-over-fulltext: let the estimate consult the FTS cache.
fts_estimate_word_docs() probed only the on-disk auxiliary INDEX_[1..6]
table. Documents inserted but not SYNCed yet are only in the in-memory
FTS cache, so they were missed entirely, and on a table that has never
been SYNCed the auxiliary table is empty and the estimate was simply
"unknown".
Look in the cache as well. fts_index_cache_t::words is an rb tree of
fts_tokenizer_word_t, and fts_node_t::doc_count already holds the number
of documents in the node's ilist, so this is one rbt_search plus a walk
over a short vector: no ilist decoding and no I/O.
Three details:
- the cache mutex is taken with trylock. This runs during
optimization, where a SYNC holding cache->lock across SQL execution
would stall the optimizer; dropping the cache contribution is the
better trade.
- nodes flagged fts_node_t::synced are skipped. An in-flight SYNC has
already written them out, and the estimator reads the B-tree without
a read view, so it sees those records; counting the node too would
count its documents twice.
- index_cache->words is NULL between fts_cache_clear() and
fts_cache_init(). The query path never observes that because it
runs after fts_init_index(); the estimate does not run it.
DB_RECORD_NOT_FOUND now means both sources are empty. The cache on its
own cannot prove a word absent, because it is only complete once
fts_init_index() has run, and an estimate must have no side effects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MDEV-40168: JSON-over-fulltext: move the estimator into fts0est.cc.
Pure code motion, no functional change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adjust the MVI tests to the estimate that reads the FTS cache
fulltext_estimate() used to see only the on-disk auxiliary table, so the
multi-valued index tests, which insert and immediately EXPLAIN, got
"unknown" for every element key: the conjunctive accesses fell back to a
guess of 10 rows and the disjunctive ones were dropped for want of a
cost. Both tests worked around that with OPTIMIZE TABLE under
innodb_optimize_fulltext_only. The estimate consults the cache now, so
the workaround is gone and the row counts in the plans are real.
Two sections of the trace test were written before the estimate existed
and no longer showed what they said they did:
- "Several element keys" printed one range, not several, because a
conjunctive access now keeps only the rarest key. It runs on a table
with a skewed distribution instead ("bbb" in every row, "aaa" in one),
which makes the choice of key visible rather than a tie, and checks
the rows against the same query with IGNORE INDEX.
- The JSON_OVERLAPS section is where several ranges are printed now, so
it says so, along with the estimates adding up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Trivial cleanups and comments
Move QUICK_MVI_SELECT into opt_multi_valued_index.cc
Keep the MVI access of a table in its JOIN_TAB
The access was looked up through JOIN::get_mvi_access_for_table(), which
indexed a per-JOIN array by table->tablenr. Nothing else about how a
table is going to be read lives there, so put it where the rest does:
JOIN_TAB::mvi_access. Mvi_context is then just the result of the WHERE
analysis - the indexes and the accesses it found - and the choice of
which access a table uses is made per table, where it belongs.
setup_mvi_access_for_table() makes that choice and marks the index in
const_keys and keys, which make_join_statistics() used to do in a loop of
its own right after update_ref_and_keys(). It runs next to
add_group_and_distinct_keys(), the other place that adds to const_keys
for something the range optimizer would not find by itself, and just
before the range analysis those bits exist for.
get_quick_record_count() takes the JOIN_TAB rather than the TABLE now,
which is all it needed the JOIN for.
Also fix the header comment of Mvi_access::estimate_records(), which
still described the old fallback for a conjunctive access that could not
be estimated at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Do the MVI analysis one table at a time
setup_mvi_quick() ran once per JOIN, from JOIN::optimize_inner(): it
collected the MV indexes of every leaf table into one list, walked the
WHERE clause once, and left the accesses it found in JOIN::mvi_ctx for
setup_mvi_access_for_table() to dig through, filtering by
access->index->vcol->table == tab->table.
Nothing about it needed to be JOIN-wide. Each JOIN_TAB now has its own
Mvi_context describing the access to its own table, and everything
setup_mvi_quick() did happens in setup_mvi_access_for_table(): collect
that table's MV indexes, analyze the condition, pick the access. The
context is only kept when there is an access to use, so the chosen one is
Mvi_context::best and JOIN_TAB has a single MVI member. The table filter
becomes a DBUG_ASSERT: ctx->indexes holds only this table's indexes and
get_mvi_index() matches the predicate against those with Item::eq(),
which compares Field pointers, so an access can only ever be on the table
whose column the predicate names - even when two tables carry identical
MVI definitions.
The condition to analyze comes from get_sargable_cond(), the same one the
range analysis of that table uses twenty lines later. For a table on the
inner side of an outer join that is the ON expression rather than the
WHERE clause, so MVI access now works there too. It is sound for the same
reason the range optimizer may do it: the scan is a necessary, not a
sufficient condition, the JSON predicate stays in the ON expression and
does the exact filtering, and outer rows that find no match are
NULL-complemented as usual. The new test checks both, against the same
queries with IGNORE INDEX.
Two consequences of the analysis no longer being a pre-pass:
- It runs on the condition the range optimizer will see, after
simplify_joins(), substitute_indexed_vcols_for_join() and
optimize_cond(), rather than on the freshly parsed WHERE.
- Const tables are not analyzed at all, the per-table loop having
skipped them before this point. They never get range analysis.
JOIN_TAB::mvi_ctx also needs no explicit reset between re-optimizations:
make_join_statistics() bzeroes the JOIN_TAB array.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make innodb_fts.estimate's clamped count deterministic
The "deleted rows are still counted" case failed about one full-suite run
in three:
-Note 1105 fulltext_estimate('gamma')= 4
+Note 1105 fulltext_estimate('gamma')= 5
The number it checks is the clamp in ha_innobase::fulltext_estimate(),
which is dict_table_get_n_rows() - the table statistics. The DELETE just
before it changes half the rows, which queues a background statistics
recalculation, and whether that has run by the time of the SELECT depends
on how loaded the machine is.
ANALYZE TABLE after the DELETE recalculates them on the spot and clears
the counter that would have triggered the background one, so the clamp
has one value to report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rename collect_mvi_vcols_for_table to collect_mvi_indexes_for_table
It collects Mv_index objects, not vcols.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keep only the chosen Mvi_access in JOIN_TAB
JOIN_TAB held the whole Mvi_context the analysis produced, but outside
setup_mvi_access_for_table() the only thing ever read out of it was
mvi_ctx->best. The rest is scratch: indexes feeds get_mvi_index(),
accesses is what the last-wins loop picks best out of, and thd is there
for the mvi_analyze() callbacks.
So JOIN_TAB keeps the access itself, and the context becomes a local of
setup_mvi_access_for_table() - which is what the TODO there asked for:
nothing is allocated for a table that has no MVI key, or whose condition
yields no access. The access outliving the context is safe because
neither it nor the Mv_index it refers to belongs to the context: both are
allocated on the MEM_ROOT, and the lists only hold link nodes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Show multi-valued indexes in SHOW CREATE TABLE
create table t1 (c int, j json,
key idx ((CAST(j->'$.tags' AS CHAR(6) ARRAY))));
printed the two columns and no index at all. Nobody had decided to hide it:
it inherited invisibility from the internal column that backs it. The
grammar makes DB_MVI_<n> INVISIBLE_FULL, init_from_binary_frm_image() turns
a hidden key part into a hidden key, and store_create_info() skips keys with
HA_INVISIBLE_KEY. Long unique hash keys - the other kind of key built over a
column the user cannot name - are already exempted from that; exempt the
multi-valued index the same way, and print it as
KEY `idx` ((cast(json_extract(`j`,'$.tags') as char(6) array)))
which is the expression it was declared with and all it takes to re-create
it. The internal column stays out of the output: it cannot be printed as a
column, since there is no syntax that would recreate the pairing.
Item_func_mvi_encode::print() cannot produce that form. Its output is what
pack_expression() puts in the FRM, and that is read back as a call of
mvi_encode(), the only form the parser accepts outside an index definition.
So the CAST spelling gets a printer of its own, sharing the type printing.
SHOW INDEX and I_S.STATISTICS list the key now too - the key part is let
through the invisibility filter - so they no longer need
debug_dbug=test_invisible_index, and the tests stop setting it (it also
injected a stray invisible1 column and key into their output).
The same HA_INVISIBLE_KEY drove mysql_prepare_alter_table(), which drops
such keys from the list of keys carried into the rebuilt table - and
INVISIBLE_FULL columns from the list of columns. So
ALTER TABLE t1 ADD COLUMN x INT;
silently dropped the index. The key survives now, and its column is carried
over with it, for exactly as long as the key lives: DROP KEY takes the
column with it, so the name is free again afterwards.
While at it, make_internal_field_name() looped forever when create_list is
empty: dup_found started at true and the loop that clears it does not run.
The MVI path is the only caller that can hit that, and it does - with
ALTER TABLE ... ADD KEY ((CAST(... ARRAY))), which used to hang the server
and now works.
A fulltext key over several arrays,
KEY idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
(CAST(j->'$.b' AS CHAR(6) ARRAY)))
has no single expression to print and no syntax of its own to be read back,
so it stays hidden, exactly as before. The optimizer still uses its parts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only allow one key part in an index over an ARRAY
create table t1 (j json, key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY)),
(CAST(j->'$.b' AS CHAR(6) ARRAY))));
was accepted without a word. Each ARRAY key part gets an internal column of
its own, and they all became key parts of one fulltext key: the tokens of
both arrays end up mixed in a single index, and the optimizer would then
search that index for the keys of one array and get the rows of the other as
well. There is also no way to show such a key, or to read one back.
init_key_part_spec() now rejects a key that has an ARRAY key part and more
than one key part, on both the CREATE TABLE and the ALTER TABLE path.
The other order, KEY idx (c,(CAST(... ARRAY))), was already rejected: the
ARRAY part makes the key FULLTEXT, and `c' cannot be part of one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A plain KEY is the only index type allowed over an ARRAY
create table t1 (j json, unique key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
create table t1 (j json, primary key ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
create table t1 (j json, fulltext key idx ((CAST(j->'$.a' AS CHAR(6) ARRAY))));
were all accepted without a word, and all produced the same thing: a plain
index. The key type the user wrote was simply overwritten with
Key::FULLTEXT, so the table ended up with no unique constraint, or no
primary key, or with a FULLTEXT index that MATCH() finds nothing in - the
index holds encoded element keys, not the text.
What the server builds for an ARRAY is a fulltext index over those encoded
elements, and it can only mean what a plain KEY means. Say so: reject any
other type, in the grammar, before that overwrite loses what was asked for.
CONSTRAINT ... UNIQUE and the ALTER TABLE forms go the same way. SPATIAL
and VECTOR are already syntax errors for an ARRAY key part; they are in the
switch anyway so it stays exhaustive.
FOREIGN KEY is unaffected: it builds its key with Key::MULTIPLE and cannot
be told apart here. It is rejected, further down, by the engine - "Foreign
key constraint is incorrectly formed".
The count of key parts is now also checked in the grammar, and not only in
init_key_part_spec(). Otherwise the first ARRAY part of a two-part key sets
the type to FULLTEXT, and the second part reports "Incorrect usage of
FULLTEXT and ARRAY" for a key nobody declared FULLTEXT. As a side effect
KEY idx (c,(CAST(... ARRAY))) now gives the same "max 1 parts" error as the
other orders, instead of ER_BAD_FT_COLUMN for `c'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
57ac7da to
2d107e1
Compare
TODOs on top of those in the patch diff: