feat(lore-0167): add prices.usd_rate and snapshot the peg rates from oracle_prices - #191
Merged
Conversation
…oracle_prices close_usd is not a stored fact, it is a cached product: every enrichment tier computes close * <USD rate of the quote asset at that time>, and the rate is a function of (quote asset, timestamp) only. Today we look it up, multiply it into hundreds of millions of rows, and discard it. This writes it down. The urgency is retention. oracle_prices is pruned at INTERVAL 13 MONTH, so the earliest depeg-aware readings age out permanently around 2026-10/11. A view cannot avoid this by joining oracle_prices directly - the published series would MUTATE as rows age out, a bucket reading 0.9993 silently reverting to a $1 fallback - which is why views.sql forbids that join. Hence a forever-retained snapshot. Keyed on natural identity, never asset_id: 0139 is confirmed genuine collisions, measured today at 3,281 ids serving 6,568 identities. The table is deliberately absent from cleanup-worker's opt-in RETENTION list, with a tripwire test - the protection is an ABSENCE, exactly the invariant a later reader breaks with one tidy line. populate_usd_rate_from_oracle copies peg observations incrementally by per-identity watermark; oracle-worker calls it after write_oracle. The call is NON-FATAL by design: oracle_prices is the source of truth and is already written, the copy is derived and self-heals on the next run, and failing here would stop oracle polling itself - trading a durable gap for a live outage. The 0139 guard is load-bearing, not decoration. oracle_prices is asset_id-keyed and usd_rate is identity-keyed, so this copy is the one place the two key spaces meet; an unchecked translation would file one asset's readings under another's identity in a table meant to be trusted forever. Refused in both directions, and the test asserts the refusal writes nothing. XLM is polled but deliberately not snapshotted - 0154 owns the pivot methods - recorded as an explicit scope boundary WITH the counter-argument that the 13-month clock applies to XLM's history identically. Tests: 2 CH ITs (copy/watermark/no-duplicate-re-run, and the 0139 refusal), a shape IT asserting the sorting key excludes asset_id, and the retention tripwire. Workspace lib suite green on the 26.3.10.60 pin; no new clippy warnings. Statement-count guard in lib.rs updated 29 -> 30.
… method in key Eight review findings, all verified against the code before accepting. Two serious. The 0139 guard ran INSIDE the write loop, so a failure on a later identity left earlier ones already written - a partial write, which is the exact failure mode the guard exists to prevent. My test could not catch it because it used a single identity. Guards now run as a pre-pass over every identity before any write, and the error names every offender. The resume watermark was max(timestamp) with a strict > filter, which silently skips any reading landing BELOW the frontier. Not hypothetical: write_oracle is also called from sdex-backfill/ingest.rs and prices-ledger-processor/reconcile.rs, which decode oracle readings from HISTORICAL ledgers. Once the scheduled worker advanced the watermark, a backdated reading would never be snapshotted and would then expire from oracle_prices at 13 months - precisely the permanent loss this table exists to prevent. Replaced with a gap-filling LEFT ANTI JOIN on (timestamp, value); both tables are small so the cost is nil. Anti-joining on the value also makes an upstream correction re-copy and win on version, which the strict > had made unreachable despite the doc claiming otherwise. method added to the sorting key: without it a 'pivot' row from 0154 at the same (identity, timestamp) as a measured 'oracle' reading would silently replace it under RMT, later-write-wins rather than better-evidence-wins. Fixed while the table is empty; changing a sorting key later means a rebuild. rates_snapshotted was computed then dropped by the Lambda entrypoint - with the deliberate non-fatal error path, a permanently broken snapshot would report success ~288 times a day with no counter moving. Now logged and returned. Also: stats count rows written rather than identities attempted; newest is per-identity so one stalled peg is visible instead of hidden behind a max(); and oracle_name is a parameter, not a hardcoded 'reflector', because the enrichment tier reads it from config and the snapshotted rate must be the rate that priced the candles or 0154 constraint 5 compares two different things. Two new ITs cover the highs. Workspace lib suite + all CH ITs green on 26.3.10.60.
Found by measuring prod, not by review. Sizing usd_rate meant querying oracle_prices, and min(timestamp) came back 1970-01-21 - which is 0086, an open confirmed bug where the oracle worker intermittently writes the real epoch divided by ~1000, with a CORRECT price and a junk timestamp. The population copied o.timestamp verbatim, filtered only on price_usd > 0, so those rows would have been snapshotted. That is strictly worse here than upstream: oracle_prices sheds them at 13 months, usd_rate is retained FOREVER, so a known defect would have become permanent history in a table whose entire selling point is being trustworthy. Adds ORACLE_EPOCH_FLOOR (2020-01-01) to the copy predicate, with a test asserting a 0086-shaped row is skipped while the good reading beside it is copied. The floor cannot exclude real data - no oracle we poll existed before Soroban - and it does NOT fix 0086, which still pollutes oracle_prices and every other reader of it.
…view Adds section 3.4a alongside oracle_prices, plus the ERD entity (both the main diagram and Appendix A), the storage-engines summary row, and the retention section. The retention entry is the one that matters most: it states that usd_rate is NEVER pruned and WHY - the cleanup list is opt-in, and this table exists precisely because oracle_prices expires and takes the earliest depeg-aware history with it, unrecoverably. Someone tidying the retention list is the realistic way that gets undone, so the reason sits next to the rule rather than only in the task. The section also records the things that are easy to get wrong later: the key is natural identity not asset_id (0139 is confirmed collisions, 3,281 ids over 6,568 identities); method is in the sorting key so a pivot estimate cannot silently replace a measured reading under RMT; absence is the signal for pre-oracle history, so no synthetic $1 rows; and the measured size (~2.3 MiB per year at worst) so nobody re-litigates forever-retention on cost grounds. clickhouse-prod-schema.sql is deliberately untouched - it documents BE's schema, not ours.
Measured live oracle readings on prod: USDC 1.00066784838102 (+0.067%), USDT 0.99930223861292 (-0.070%). Three properties worth having on the record. The ~0.1% figure is real and ordinary, not a depeg event. The two deviate in OPPOSITE directions, so the spread between them is ~0.137% and a flat $1 does not mostly cancel - anything comparing a USDC-denominated value against a USDT-denominated one carries the full 0.14%. And it is a persistent bias rather than noise: five consecutive 5-minute readings held sign and magnitude to four decimal places, so it does not average out across candles the way jitter would. That is a stronger argument for 0168 than a depeg would be, because it is permanent and invisible rather than rare and dramatic. The gap: 0168 fixes the VIEW's peg fallback, while the enrichment peg tier bakes the same flat $1 into close_usd itself for every USDC/USDT-quoted candle the oracle tier did not reach - all deep history plus anything outside the staleness bound. Shipping 0168 leaves the view and the candles disagreeing by that margin. Not folded in: pointing the peg tier at usd_rate is a write-path change to the enrichment hot loop that 0111 owns, and correcting history means re-enrichment rather than a view swap. Also flagged not to queue it ahead of 0172 on magnitude - 0.07% on close_usd is plausibly fine, 0172 is a ~7x error on 102 live pools.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
close_usdis not a stored fact — it's a cached product. Every enrichment tier computesclose × <USD rate of the quote asset at that time>, and the rate is a function of(quote asset, timestamp)only, never of the candle being priced. Today we look it up, multiply it into hundreds of millions of rows, and discard it. This writes it down.⏳ Why now
oracle_pricesis pruned atINTERVAL 13 MONTH, so the earliest depeg-aware readings age out permanently around 2026-10/11. This is the only open task whose input data expires.A view can't dodge it by joining
oracle_pricesdirectly: the published series would mutate as rows age out — a bucket reading0.9993silently reverting to a$1fallback later — which is exactly whyviews.sqlforbids that join. Hence a forever-retained snapshot.Key is natural identity, never
asset_id0139 is confirmed genuine collisions, measured today at 3,281 ids serving 6,568 identities. An
asset_idkey would be non-unique by construction.The 0139 guard is the load-bearing piece
oracle_pricesisasset_id-keyed andusd_rateis identity-keyed, so this copy is the one place the two key spaces meet. An unchecked translation would file one asset's oracle readings under another asset's identity — silently, in a table built to be trusted forever. The write is refused in both directions, and the test asserts the refusal writes nothing (a partial write is the failure mode the guard exists to prevent).Two deliberate design calls
The snapshot is non-fatal in the worker.
oracle_pricesis the source of truth and is already written by that point; the copy is derived and watermarked, so a failed pass self-heals next run. Failing the worker would stop oracle polling — trading a durable, self-healing gap for a live outage.Retention protection is an absence, so it gets a tripwire.
RETENTIONis an opt-in allowlist;usd_rateis absent and must stay absent. That's precisely the invariant someone breaks with one tidy-looking line, socleanup-workernow has a test whose failure message explains the consequence.Scope boundary flagged, not silently taken
XLM is polled but not snapshotted — 0154 owns the
pivotmethods. Recorded onpeg_identities()with the counter-argument: the 13-month clock applies to XLM's history identically, so if 0154 hasn't started before202509ages out, this should be reconsidered rather than deferred by default.Tests
2 new CH ITs on the 26.3.10.60 pin (copy → re-run without duplication → incremental append; and the 0139 refusal), a shape IT asserting the sorting key excludes
asset_id, and the retention tripwire. Workspace lib suite green, no new clippy warnings, statement-count guard updated 29 → 30.Two self-inflicted bugs are recorded in the task file because they were invisible in review:
watermark_beforedefaulted to0with a.min()that pinned it there forever, and the two ITs shared the realpricesdatabase and truncated each other under cargo's parallel runner — both failed in ways that looked like product bugs.Merging changes nothing on prod. The population runs when
oracle-workernext executes, and 0154 constraint 5 — reproduce today'sclose_usdfrom the stored rate — is still open and gates any consumer trusting the table. Query is in the task file, with a warning not to reconcile through USDT until 0172 is understood.