Skip to content

docs(lore-0111): add rate-driven discovery as option 5 - #193

Merged
karczuRF merged 1 commit into
developfrom
docs/0111_rate-driven-discovery-option
Aug 10, 2026
Merged

docs(lore-0111): add rate-driven discovery as option 5#193
karczuRF merged 1 commit into
developfrom
docs/0111_rate-driven-discovery-option

Conversation

@karczuRF

Copy link
Copy Markdown
Collaborator

0111's options list predates prices.usd_rate existing, so it never considered the cheapest form of option 3.

The reframing

The defect isn't that the arithmetic is expensive — it's that discovering the work costs a full scan. Enrichment asks "which candles are missing a price?", which is only answerable by reading all 545M of them, twice per batch.

With rates stored as first-class rows it can ask "which rates are new since I last ran?" — a small query against a small table — then resolve a bounded candle set. Same insight as option 3's pending-work table, but using a queue we're already building for other reasons.

The two aren't exclusive: a rate watermark covers "a new rate arrived", an ingest queue covers "a new candle arrived", and enrichment needs both edges.

Recorded with its limits, so it isn't mistaken for a shortcut

  • A new rate doesn't name the candles that need it. You still resolve "candles quoted in X over window W" — and quote_asset_id is only the 2nd sort-key column, so that lookup has no clean index path. That's the same projection-cost unknown 0151 flagged and 0167 deliberately didn't pull forward.
  • It does nothing for the existing backlog. A queue helps from when you start queueing; accumulated zeros still need one bounded sweep (option 1).
  • ⚠️ It cannot be used to skip 0111. Full rate coverage needs 0154's pivot tiers, and 0154 is hard-blocked behind 0111. 0167 landed only the peg subset (USDC/USDT, method='oracle'). So this sequences with 0111, not instead of it.

And the misreading to avoid

⚠️ This is not "compute close_usd from usd_rate at read time". That's the schema-wide refactor 0151 rejected — and it wouldn't even be cheap: the pivot rate is itself derived from candles (the XLM/USDC vwap), so the scan moves rather than disappears, and the cost lands on every read of a surface BE already calls slow at 4.6 s.

close_usd stays stored, non-nullable, written in place.

The options list predates prices.usd_rate existing, so it never considered the
cheapest form of option 3.

The defect is not that the arithmetic is expensive - it is that DISCOVERING the
work costs a full scan. Enrichment asks 'which candles are missing a price?',
answerable only by reading all 545M of them. With rates stored as first-class
rows it can ask 'which rates are new since I last ran?' against a small table,
then resolve a bounded candle set.

Recorded with its three limits so it is not mistaken for a shortcut: a new rate
does not name the candles that need it, and quote_asset_id is only the 2nd
sort-key column so that lookup has no clean index path; it does nothing for the
existing backlog; and it cannot be used to SKIP 0111, because full rate coverage
needs 0154's pivot tiers and 0154 is hard-blocked behind 0111. 0167 landed only
the peg subset.

Also flagged the misreading to avoid: this is not 'compute close_usd from
usd_rate at read time'. That is the refactor 0151 rejected, and it would not even
be cheap - the pivot rate is itself derived from candles, so the scan moves
rather than disappears, and the cost lands on every read of a surface BE already
calls slow.
@karczuRF
karczuRF merged commit cfea47d into develop Aug 10, 2026
3 checks passed
@karczuRF
karczuRF deleted the docs/0111_rate-driven-discovery-option branch August 10, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant