docs(lore-0111): add rate-driven discovery as option 5 - #193
Merged
Conversation
The options list predates prices.usd_rate existing, so it never considered the cheapest form of option 3. The defect is not that the arithmetic is expensive - it is that DISCOVERING the work costs a full scan. Enrichment asks 'which candles are missing a price?', answerable only by reading all 545M of them. With rates stored as first-class rows it can ask 'which rates are new since I last ran?' against a small table, then resolve a bounded candle set. Recorded with its three limits so it is not mistaken for a shortcut: a new rate does not name the candles that need it, and quote_asset_id is only the 2nd sort-key column so that lookup has no clean index path; it does nothing for the existing backlog; and it cannot be used to SKIP 0111, because full rate coverage needs 0154's pivot tiers and 0154 is hard-blocked behind 0111. 0167 landed only the peg subset. Also flagged the misreading to avoid: this is not 'compute close_usd from usd_rate at read time'. That is the refactor 0151 rejected, and it would not even be cheap - the pivot rate is itself derived from candles, so the scan moves rather than disappears, and the cost lands on every read of a surface BE already calls slow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0111's options list predates
prices.usd_rateexisting, so it never considered the cheapest form of option 3.The reframing
The defect isn't that the arithmetic is expensive — it's that discovering the work costs a full scan. Enrichment asks "which candles are missing a price?", which is only answerable by reading all 545M of them, twice per batch.
With rates stored as first-class rows it can ask "which rates are new since I last ran?" — a small query against a small table — then resolve a bounded candle set. Same insight as option 3's pending-work table, but using a queue we're already building for other reasons.
The two aren't exclusive: a rate watermark covers "a new rate arrived", an ingest queue covers "a new candle arrived", and enrichment needs both edges.
Recorded with its limits, so it isn't mistaken for a shortcut
quote_asset_idis only the 2nd sort-key column, so that lookup has no clean index path. That's the same projection-cost unknown 0151 flagged and 0167 deliberately didn't pull forward.method='oracle'). So this sequences with 0111, not instead of it.And the misreading to avoid
close_usdfromusd_rateat read time". That's the schema-wide refactor 0151 rejected — and it wouldn't even be cheap: the pivot rate is itself derived from candles (the XLM/USDC vwap), so the scan moves rather than disappears, and the cost lands on every read of a surface BE already calls slow at 4.6 s.close_usdstays stored, non-nullable, written in place.