MDEV-32286 Reuse remembered clustered leaves in secondary-index scans

[iMineLink]

Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row whose clustered-index record a secondary-index
scan must read, although consecutive rows often land on the same clustered
leaf page. A non-covering scan needs one for every row, a locking read needs
one whatever the secondary index holds, because an exclusive select lock
type makes ha_innobase::build_template() build its template against the
clustered index, and a covering scan needs one for every row of a secondary
leaf whose PAGE_MAX_TRX_ID its read view cannot see. ANALYZE FORMAT=JSON
charges each descent its full height, and those descents are nearly the
whole cost: the secondary index is charged its own descent and one page for
each further leaf, and nothing per row, because the position that its cursor
holds between two rows is restored optimistically, which latches the leaf
again without counting an access. The range estimate that
ha_innobase::records_in_range() makes for the optimizer is charged over the
same leaves, once more. So pages_accessed is the row count times the height
of the clustered index, plus that handful. 1000 rows over a 2-level
clustered index cost 2006, of which 3 are the scan of the secondary index
and 3 the range estimate, and 750 rows over a 3-level one cost 2291, of
which 27 and 14, where a full table scan of the same data costs 23 and 110.

Remember, in the new row_prebuilt_t::clust_leaf_hint, the
CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this
statement reached, most recently used first. Each slot names one leaf: its
page number, copies of its first and last user record truncated to the key
fields, which bound the key range the leaf held when it was remembered, and
the rec_get_offsets() of both. The copies are needed because the page is
unlatched between two lookups, and the offsets spare a lookup the parsing of
them. Several slots serve the scans that alternate between a few leaves,
which one slot cannot serve at all, and a descent refreshes the slot of a
leaf that is remembered already rather than spend a second one on the same
page. A slot owns its key buffers and grows them only when a longer key
arrives, so a row allocates nothing. The used-slot count and the miss
counter are reset per statement in ha_innobase::reset(), matching
autoinc_last_value.

A lookup first compares its key against the remembered ranges, so an
uncorrelated scan settles its misses in memory, with no buffer pool access
and no pages_accessed. Only a covering range is probed, through the new
btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get():
a hint is never derived from a latched parent page, so by the time it is
tried it may precede the caller's already-latched secondary-index leaf in
the latching order, where a blocking wait can deadlock. A stale range costs
a wasted probe or a needless descent, never a wrong result, because the
checks that try_leaf_hint() makes on the latched page remain the sole
authority, and the ranges therefore need no invalidation protocol.

After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan
gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of
the slots and the copies that refresh them, which are the larger half of
their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count
again, so a scan whose order becomes correlated only later recovers, and the
trial that this begins refreshes the slots as it goes.

The run is short because a hit saves little where the pages above the leaf
are resident: one buffer pool access and one page-local search for each
level. Measured against the same tree built without the hints, at 16k with a
resident working set and no adaptive hash index, a wholly correlated scan
runs a quarter faster over half the page accesses, a scan that answers three
lookups in five runs level with it over 30% fewer, one whose locality
appears only half way through runs an eighth faster over a quarter fewer,
and a scan that answers nothing stays within the noise. A run of 8 is what
keeps that last one there. A clustered index small enough to stay in cache
is the exception that the run does not catch: it answers often, so the count
never builds, and it saves nothing, because the descent that a hit replaces
costs almost nothing there. Such a scan pays about a tenth.

Where the adaptive hash index is enabled, the hints are neither used nor
collected: its guess solves the same problem better, landing on the record
with no page-local search and no page access to charge. It is off by
default, so the hints are active in a default configuration.

innodb.non_covering_sec_idx_scan measures pages_accessed over key orders
that differ in how closely the secondary order tracks the clustered one, and
eight further tables check query results over the record formats and key
shapes that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE, to leaves that split and merge while a
locking read walks them, and to a record that a remembered leaf supplies for
a scan that must then rebuild an older version of it.
non_covering_sec_idx_scan_debug runs the same body with the hints turned
off, through a debug switch that returns before a lookup tests or refreshes
the slots, so a diff of the two .result files is what the hints save: 2006
to 1028 (2-level clustered index), 2291 to 1007 (3-level), 20020 to 15780
(decorrelated), 4006 to 2012 (two interleaved key ranges), 20020 to 20006
(shuffled), 12016 to 9289 (locality in the second half alone) and 20020 to
10042 for a covering scan that FOR UPDATE makes non-covering, where the same
scan without FOR UPDATE costs 20 in both files. The locality count pins the
retry: without it the scan would keep only the 82 hits it makes before it
gives up, and the count would be 11934.
main.rowid_filter_innodb: 90 to 81, and its ahi combination unchanged.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Thirunarayanan]

row_sel_clust_leaf_hint_covers() discards a slot whose n_core_fields no
longer matches the index, because a delete that empties a single-page table
invokes dict_index_t::clear_instant_alter(). Are we have any test case for it?

[Thirunarayanan]

noexcept is missing?

[Thirunarayanan]

The commit states that a clustered index small enough to stay in cache "pays
about a tenth". What if the query does only one lookup of clustered index? (pointed query?)
we may end up with 2 * rec_get_offsets(), 2 * ut_malloc(), 2 * rec_copy_prefix_to_buf()?
what we cache it from 2nd record ?
In remember(), do we something if (prebuilt->clust_leaf_hint_miss < 2) return; or something?

[Thirunarayanan]

Also avoid when clustered index height is 0? (I mean only root page exist?)

[Thirunarayanan]

can we compute this one first before last, last_offs?
if (hint.rightmost == false) {
calculate the hint.last, hint.last_offs;
}

[Thirunarayanan]

record prefix is bounded by page size. so can we make last_buf_size, first_buf_size to uint32_t?
Rearrange the fields and try to reduce clust_leaf_hint_t size.

[Thirunarayanan]

what if the query does keep juggling between 2 pk pages? Are we moving the bytes un-necessarily ?
If you think it is necessary then keep separate variable MRU in prebuilt. In that case, you may move 3 bytes only.

[Thirunarayanan]

we already going through per-field offsets in rec_copy_prefix_to_buf(). Again we do the same in rec_get_offsets()

[Thirunarayanan]

page_eviction, s_lock_try() fails to concurrent writer or no uncompressed frame at all. Are we missing this message?

[Thirunarayanan]

Why 4? not 2?

[Thirunarayanan]

Use parallel array? first[4], last[4] etc... Because covers[] won't touch first_buf, first_buf_size, last_buf, last_buf_size at all. Just an idea though