Skip to content

feat(0157): default key limits of 1 req/s + 100k monthly quota #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adamkoot merged 9 commits into from
Aug 12, 2026
Merged

feat(0157): default key limits of 1 req/s + 100k monthly quota #195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
936a091
chore(lore-0157): activate task
adamkoot Aug 10, 2026
d4ad060
feat(lore-0157)!: cut default key limits to 1 req/s + 100k/month
adamkoot Aug 11, 2026
84dc267
docs(lore-0157): retag 429s as monthly quota; add manual-tier runbook
adamkoot Aug 11, 2026
22abb57
docs(lore-0157): record the one-plan outcome and its corrections
adamkoot Aug 11, 2026
308c1ad
refactor(lore-0157)!: name the plan family pricing-api-free
adamkoot Aug 12, 2026
831ce5b
docs(lore-0157): harden the manual-tier runbook and finish the 429 sweep
adamkoot Aug 12, 2026
30c501c
docs(lore-0157): close the last blast-radius question on the key rota…
adamkoot Aug 12, 2026
51e86f5
docs(lore-0157): fix the contradictory throttle comment
adamkoot Aug 12, 2026
a3e8c56
docs(lore-0157): warn that broadening is silent on deletion
adamkoot Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/database-schema/database-schema-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1797,7 +1797,7 @@ asset's inception date, `GET /assets/{asset_identifier}/ohlcv` includes a
| Layer | Strategy |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **API Gateway caching** | Built-in response cache (0.5 GB). Per-endpoint TTLs: `/assets` list 60s, `/ohlcv` 60s, `/price` 15s, `/backfill/status` 30s. Cache key includes query params. `POST /prices/batch` uncached |
| **API Gateway throttling** | Request throttling (100/s per API key, 1000/s global burst) |
| **API Gateway throttling** | Request throttling (1 req/s sustained, burst 5, 100 000 req/month per self-service key — task 0157; 200 req/s per method stage-wide) |
| **Lambda** | Rust binary with `lambda_runtime`. Sub-millisecond cold starts. Stateless, auto-scales to concurrency limit. No VPC, so no ENI provisioning latency on cold start |
| **ClickHouse client (`clickhouse` crate)** | Warm connection pool reused across Lambda invocations to amortise mTLS handshake (~80-130 ms cross-cloud RTT to Caddy). Per-request payloads batched per-ledger so a typical invocation issues 1–2 INSERTs, not one per trade |
| **Sort key + partitioning** | Per-granularity tables sorted by `(asset_id, quote_asset_id, source, timestamp)`; monthly partitions on `timestamp`. Partition pruning + sort-key skip eliminate irrelevant months and assets on hot reads |
Expand All @@ -1811,13 +1811,13 @@ instance on a single Hetzner box behind Caddy:443). Prices-api joins as a
second tenant via its own `prices` database, isolated by ClickHouse's native
multi-tenant primitives (database, user, quota, profile).

| Metric | Value | Source |
| ---------------------------- | --------------------------------------------------------------------------- | -------------------------- |
| Prices-api storage footprint | **~3.5-6 GB/year** (realistic, retention-amortised) | Tasks 0060 + 0063 measured |
| Average per-ledger storage | **~1.9-3.7 KB/ledger** (activity-dependent, ~2× spread) | Tasks 0060 + 0063 measured |
| Strongest size lever | Retention-cap `_1h`/`_4h` → bounds DB at ~9 GB @ 10yr (vs ~43 GB unbounded) | Task 0060 measured |
| Write rate | ~1 INSERT per ledger (~12k/day per env at mainnet cadence) | §6.1 |
| Read rate | API-Gateway-throttled ≤100 req/s per key, cached at gateway | §8.2 |
| Metric | Value | Source |
| ---------------------------- | --------------------------------------------------------------------------------- | -------------------------- |
| Prices-api storage footprint | **~3.5-6 GB/year** (realistic, retention-amortised) | Tasks 0060 + 0063 measured |
| Average per-ledger storage | **~1.9-3.7 KB/ledger** (activity-dependent, ~2× spread) | Tasks 0060 + 0063 measured |
| Strongest size lever | Retention-cap `_1h`/`_4h` → bounds DB at ~9 GB @ 10yr (vs ~43 GB unbounded) | Task 0060 measured |
| Write rate | ~1 INSERT per ledger (~12k/day per env at mainnet cadence) | §6.1 |
| Read rate | ≤1 req/s per key (task 0157); ≤200 req/s per method stage-wide, cached at gateway | §8.2 |

> **Sizing superseded (2026-06-19).** The original ~74 B/ledger / ~0.45 GB/yr
> figure was the task-0046 _per-event estimate_. Three ground-truth backfill
Expand Down
11 changes: 8 additions & 3 deletions docs/prices-api-general-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ to their own infrastructure at any time if needed.
| **Lambda — Asset Discovery** | Asset registry | EventBridge rate(1 hour). Detects new SEP-41 contract deployments and classic asset issuances; UPSERTs into `prices.assets` |
| **Lambda — Cleanup Worker** | Data retention | EventBridge cron(02:00 UTC daily). `ALTER TABLE … DROP PARTITION` on old monthly partitions of each per-granularity OHLCV table |
| **Lambda — API handlers** | Public API | Individual functions per route group. Rust / axum via `lambda_runtime`, 256–512 MB, 15s timeout. No VPC; outbound HTTPS-mTLS to Caddy:443 |
| **API Gateway** | Public API entry point | REST API, usage plans, API key auth, rate limiting (100 req/s per key), request validation. Built-in response cache (0.5 GB) with per-endpoint TTLs |
| **API Gateway** | Public API entry point | REST API, usage plans, API key auth, rate limiting (1 req/s sustained, burst 5, 100 000 req/month per self-service key — task 0157), request validation. Built-in response cache (0.5 GB) with per-endpoint TTLs |
| **EventBridge Scheduler** | Scheduled triggers | Cron/rate rules for all periodic Lambda workers |
| **Secrets Manager** | Credentials & mTLS material | Per-env client `{cert,key,ca}` for Caddy:443 mTLS (single JSON bundle secret per identity, named by `MTLS_SECRET_NAME`); Soroswap/Aquarius API keys; oracle contract address |
| **CloudWatch + X-Ray** | Observability | API latency, error rates, ingestion lag, Lambda duration/concurrency, backfill progress; mTLS cert NotAfter alarm |
Expand Down Expand Up @@ -1093,7 +1093,7 @@ CLI progress via direct SQL on the local workstation ClickHouse.
| Layer | Strategy |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **API Gateway caching** | Built-in response cache (0.5 GB). Per-endpoint TTLs: `/assets` list 60s, `/ohlcv` 60s, `/price` 15s, `/backfill/status` 30s. Cache key includes query params. POST `/prices/batch` uncached |
| **API Gateway throttling** | Request throttling (100/s per API key, 1000/s global burst) |
| **API Gateway throttling** | Request throttling (1 req/s sustained, burst 5, 100 000 req/month per self-service key — task 0157; 200 req/s per method stage-wide) |
| **Lambda** | Rust binary with `lambda_runtime`. Sub-millisecond cold starts. Stateless, auto-scales to concurrency limit. No VPC, so no ENI provisioning latency on cold start |
| **Database client (`clickhouse` crate)** | Warm connection pool reused across Lambda invocations to amortise mTLS handshake (~80-130 ms cross-cloud RTT to Caddy). Per-request payloads batched per-ledger so a typical invocation issues 1–2 INSERTs, not one per trade |
| **Sort key & partitioning** | Per-granularity tables sorted by `(asset_id, quote_asset_id, source, timestamp)`; monthly partitions on `timestamp`. Partition pruning + sort-key skip eliminate irrelevant months and assets on hot reads |
Expand Down Expand Up @@ -1275,6 +1275,10 @@ for the full local-CLI metrics.
1. All 7 endpoint groups return correct, schema-valid responses for at least 20 major assets
2. Load test (k6 or Locust, script provided): 100 req/s sustained for 5 minutes on
`GET /assets/{id}/price` → p95 latency <200ms, error rate <0.1%
— the target is unchanged, but since task 0157 no key in the account can sustain
it: the default plan is 1 req/s (§6). The run needs a usage plan created for it,
per `docs/runbooks/manual-api-key-tier.md`; the report must state which plan the
key was on (task 0121)
3. Cache confirmed: consecutive identical requests within TTL window return `X-Cache: Hit` header
4. VWAP calculation verifiable against raw `price_ohlcv` rows for at least 3 assets
5. `GET /backfill/status` shows `earliest_data_available` ≤ 2022-01-01
Expand Down Expand Up @@ -1332,7 +1336,8 @@ post-delivery monitoring.
2. OpenAPI spec passes `openapi-validator` lint with no errors; Swagger UI deployed
3. Onboarding portal accessible; self-service API key request flow functional
4. Integration test suite: all tests pass on CI (GitHub Actions link provided)
5. Load test report: p95 <100ms at 100 req/s confirmed
5. Load test report: p95 <100ms at 100 req/s confirmed — same caveat as Tranche 2
AC 2: a purpose-built usage plan is required, and the report names it
6. Security checklist signed off: no wildcard IAM, ClickHouse endpoint reachable only via
mTLS through Caddy:443, mTLS cert + key in Secrets Manager (not env vars), all inputs
validated
Expand Down
Loading
Loading