SQL-150: durable temporary objects - #37878
Draft
SangJunBak wants to merge 15 commits into
Draft
Conversation
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
3 times, most recently
from
July 27, 2026 21:10
113a967 to
de93fc8
Compare
SangJunBak
changed the title
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
3 times, most recently
from
August 5, 2026 00:55
71cf04f to
c395227
Compare
Initially the design doc stated that we wanted to persist sessions in the Catalog. Through benchmarking, it was found to create majoir regressions on CPS, even with optimizations. Thus we keep mz_sessions as a builtin table but will continue to persist durable objects.
Generic catalog migration version bump. Copies everything and is intended to make the review easier.
Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID and parented to the temporary schema sentinel id (u0). The in-memory TemporaryItem side-channel is deleted: temp items flow through the normal durable update pipeline. On apply, an ephemeral item is routed into the owning connection's in-memory temporary schema, resolved through a session-UUID-to- connection mapping that the coordinator registers at connect and removes at terminate, strictly after the transaction dropping the session's temporary items has been applied. Applying an ephemeral item whose owner is not a session served by this process is a no-op, which covers following the catalog read-only during zero-downtime deployments, where the leader's temporary items arrive but their sessions live elsewhere. Session close drops the session's temporary items, dependents included, in one catalog transaction, before the mz_sessions retraction is queued, so a crash in between leaves a session row without items rather than orphaned items. Opening the catalog with write intent reclaims all ephemeral items (added previously), covering crashes and kill -9. Orphaned temp-table shards like mz_sessions are finalized by the existing storage metadata reconciliation. BootstrapStateUpdateKind is deleted: it existed only to exclude the TemporaryItem variant, and memory StateUpdateKind now serves both uses. Verified manually: temp create/insert/query/cascade-drop, same temp table name in two concurrent sessions, graceful-close cleanup, kill -9 + restart reclamation, DISCARD ALL, temporary_objects.slt. adapter: skip applying ephemeral item updates for non-local sessions (SQL-150) adapter: register ephemeral owner lazily at first temp-item creation (SQL-150) Registering the session uuid <-> conn_id mapping in handle_startup and unregistering it unconditionally in handle_terminate forced an Arc::make_mut clone of the Catalog on every connect and disconnect, defeating the existing gate that avoids the clone for sessions that never create temporary objects. Register the mapping in catalog_transact_inner instead, when a transaction first creates a temporary item for the session. The Arc::make_mut there is already unconditional, so registration is free. Gate unregistration at terminate on the registration existing, mirroring the temporary schema gate. Sessions that never touch temporary objects now cost zero catalog clones. The maps and methods are named after the durable ephemeral_owner_session field they resolve (ephemeral_owner_conns_by_uuid and ephemeral_owner_uuids_by_conn, with register_ephemeral_owner, unregister_ephemeral_owner, and is_ephemeral_owner), since they cover only sessions that own temporary items. The Op::CreateItem temporary branch now fails loudly when the owner is not registered, since the apply pass would otherwise silently skip the addition as non-local and diverge from the durable catalog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With temporary items durable in the catalog shard, mz_tables and mz_views can be derived from mz_internal.mz_catalog_raw instead of being written by the coordinator. Both show every item including temporary ones, matching the previous builtin tables. Temporary rows keep the temporary schema sentinel "0" in schema_id. Builtin tables and views are reported through two new generated constant views, mz_internal.mz_builtin_tables and mz_internal.mz_builtin_views, following the mz_builtin_materialized_views pattern. mz_builtin_views lists every builtin view except itself, since its definition cannot contain itself, so that one (new) view is absent from mz_views and relations derived from it. The declared RelationDesc keys of the generated views must exactly match the keys the optimizer derives from their VALUES lists (verify_builtin_descs enforces this), so both declare the keys that hold for the current builtin sets, with a NOTE on what to do when a future builtin breaks one. parse_catalog_create_sql now exposes 'definition' for views and 'source_id' for tables created from sources. The old mz_tables and mz_views shards are released through builtin schema migration replacement steps and finalized by the storage layer, like the other builtin table conversions. Verified manually: boot, user/temp tables and views appear with correct schema ids and definitions, pg_tables/pg_views joins, builtin arms populated. SLT: information_schema_tables, oid, temporary_objects, autogenerated/mz_internal.
Implementation log for the SQL-150 work: exploration facts, decisions, stage progress, and the pivot plan that moved sessions back out of the durable catalog. Not needed to understand the code changes. Drop this change before merging if the notes should not land in main.
mz_tables and mz_views select rows by parse_catalog_create_sql(...)->>'type' and read 'definition' and 'source_id' out of the same call, but the function had no tests. Pins the reported type for every statement kind an Item record can hold, the exact mz_views.definition rendering (which pg_views exposes and which moved here out of the deleted pack_view_update), its idempotence under re-parsing, source_id presence for CREATE TABLE FROM SOURCE and absence for plain and webhook tables, and the four error paths. The error paths matter more than they used to: the MVs call this inside their WHERE clause, so an item the parser rejects makes the whole relation unreadable rather than breaking one row's packing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mz_tables and mz_views became BuiltinMaterializedViews over mz_internal.mz_catalog_raw, deriving every column from durable catalog JSON, but the conversion added no test file. Every earlier conversion (mz_indexes, mz_audit_events, mz_postgres_sources) got one. Follows the established lockdown shape: one section per union branch, plus the temporary-item sentinel schema id, the exactly-once property across the user and builtin branches, and the ASSERT NOT NULL columns. Two checks are independent of the MV rather than golden values: oid is compared against a regclass cast, which resolves through the in-memory catalog, and mz_views.definition is compared against the definition of a view planned from it, so the rendering is verified to be a fixed point end to end. create_sql cannot be compared against SHOW CREATE, which humanizes item ids and pretty-prints by design. Also pins the one known difference from the old builtin tables: mz_builtin_views cannot list itself, so it is the single builtin view absent from mz_views. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Temporary items are durable items tagged with the UUID of the session that created them. Neither of the two mechanisms holding that together had any coverage. read-write: name uniqueness is scoped by the owning session, so two sessions can each hold a 'tt' in the sentinel temporary schema while one session cannot hold it twice. remove_ephemeral_items then reclaims all of them and leaves normal items alone, which matters because an over-broad filter there would silently delete real user items. open: a writable open reclaims a temporary item left behind by a process that died without closing its session, which is the only thing between a kill -9 and a permanently leaked catalog item. A read-only open must not, since a zero-downtime follower reads the leader's catalog while the leader's sessions are still live and still own theirs. The read-only case is checked before the writable one, so the ordering makes both directions observable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…150) Temporary tables and views are durable catalog items tagged with their owning session, so they need cleanup on both paths out of a session, and neither path had a test. The only existing temp-teardown coverage is DISCARD TEMP / DISCARD ALL, which is a different code path. One workflow covers both halves because they check each other. The graceful close is only meaningful if a second session's identically-named items survive it, and that surviving session is what the crash half then needs to hold open. mzcompose rather than sqllogictest or testdrive: sqllogictest cannot disconnect a session and its reset-server directive builds a fresh EnvironmentId, so it gets a new catalog rather than a restart; testdrive has no process-lifecycle action and its default connection is created once and never reconnected. Only a real SIGKILL guarantees the session-close hook did not run, which is what makes the final absence assertion evidence that boot-time reclamation happened rather than something that could pass either way. Asserts on mz_catalog_raw as well as mz_tables and mz_views, since only the raw view shows that the durable rows themselves are gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment justified the no-persistent-dependency-on-temporary rule with "temporary objects live only in the in-memory, session-scoped catalog and are never persisted", which stopped being true when temporary items became durable catalog items. The rule and the test are unchanged. Only the reason is: a temporary item is durable but owned by its session, so a persistent item referencing one dangles as soon as that session goes away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t (SQL-150) After a kill -9, ephemeral-item reclamation at catalog open deleted only the durable item rows. The graceful drop path additionally removes each temp table's storage collection metadata rows and enqueues its shard in the unfinalized_shards finalization WAL, in the same catalog commit. Because bootstrap only ever inserts metadata for collections present in the catalog and shard finalization is driven solely by that WAL, an orphaned metadata row was kept forever: the persist shard was never finalized and every boot warmed its state pointlessly. Extend remove_ephemeral_items to mirror the graceful drop within the same transaction: delete the collection metadata rows for all of each item's global ids (root and extra versions, which share one shard), enqueue unreferenced shards for finalization, and drop the items' comments (item ids are reused, so a dangling comment could re-attach to an unrelated later object and already trips the catalog consistency check) and source references (defensive, sources cannot be temporary). Finalizing the shard while its txns-shard registration dangles (nothing forgets it after a crash) is safe: every txn-wal write path to a data shard tolerates a finalized shard by early-returning, and forget skips unregistered ids. The dangling registration is a small pre-existing leak, also reachable via the graceful path's commit-to-forget crash window, and is not addressed here. Rows leaked before this fix are also not swept, since reclamation only sees current ephemeral items. Verified end to end: CREATE TEMP TABLE, kill -9 environmentd, restart. The orphaned shard shows up in initializing finalizable_shards and is finalized by the background task within seconds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> test: assert storage reclamation in the temporary item cleanup workflow (SQL-150) The kill -9 half of workflow_temporary_item_cleanup only asserted that the durable item rows were reclaimed at boot. Extend it to cover the storage side end to end: capture the surviving session's temp table shard before the kill, then assert after the restart that its StorageCollectionMetadata row is gone, that the shard was enqueued in the unfinalized_shards finalization WAL, and, via the coordinator dump, that the background task actually finalizes it. Also comment on the temp table before the kill and assert the durable comment row is reclaimed with the item. The assertions read mz_internal.mz_catalog_raw, which is the catalog shard itself, so they observe the durable rows rather than any derived relation. The WAL row is pruned again by the next committed catalog transaction after finalization, so the checks run before any post-restart DDL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
from
August 5, 2026 21:16
c395227 to
bb38386
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design doc for making temporary tables and views durable in the catalog
shard, so that mz_tables and mz_views can become materialized views over
mz_internal.mz_catalog_raw and temporary state survives in a multi-envd
world. Records why sessions stay in the mz_sessions builtin table
instead of becoming durable catalog records (connect latency coupling to
the catalog compare-and-append, churn contention with DDL on the
catalog single writer), and distills that lesson into the adapter guide.Remove these sections if your commit already has a good description!
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
Nightly: https://buildkite.com/materialize/nightly/builds/17699