feat(blockchain): log unconfigured clients periodically for persistent alerting#4235
feat(blockchain): log unconfigured clients periodically for persistent alerting#4235Danswar wants to merge 2 commits into
Conversation
|
b6ea735 to
08f81c7
Compare
…t alerting api#3878 and api#4203 downgraded an unset Tatum API key to a warn-once- per-boot skip, so a lost prod secret only leaves a single boot-time log line and the infra alert auto-resolves after its lookback window even if the key is still missing (DFXServer/server#753). BlockchainConfigCheckService now re-logs the unconfigured state every 5 minutes on prd (chain names only, no secret values), giving the alert a continuous signal that stops only when the config is restored. Gated to prd since unconfigured clients are the intended state elsewhere, with an own Process flag as kill-switch.
08f81c7 to
d8e039d
Compare
|
Reviewed. The heartbeat itself is fine — 5-min interval with jitter, prd-only guard with an early return before any registry call, the One thing I'd fix before merging, though: the blanket Two sibling services already handle this explicitly: |
The periodic config check caught every error, including a real TypeError when getClient returns a null/undefined client (e.g. an unset node URL for Bitcoin/Firo) — silently dropping exactly the chain it should surface. Check for a falsy client and warn (matching log-job/payment-balance) before reading isConfigured, and narrow the catch to the "No service found" case so genuine errors stay visible.
|
Pushed |
|
A deeper full-PR pass surfaced a few coverage limitations worth your call — these are about which chains the sweep can actually see (feature scope), not bugs in the heartbeat itself:
The null-client warn fix ( |
Why
#3878 (Cardano) and #4203 (Solana) deliberately downgrade an unset Tatum API key from a recurring ERROR to a warn-once-per-boot skip, because dfxdev runs keyless by design. DFXServer/server#730 turned that boot warn into a prod alert, but since the warn is logged only once per boot, the alert's 24h Loki window auto-resolves even if the key is still missing — a resolve doesn't mean the config was restored (DFXServer/server#753).
What
New
BlockchainConfigCheckService(blockchain shared services):BlockchainRegistryServicefor clients reportingisConfigured === falseand logs one warn line:Blockchain clients not configured: <chains>— chain names only, no secret values.prd: on dev/loc an unconfigured Tatum client is the intended state, and a periodic warn there would be permanent noise.Process.BLOCKCHAIN_CONFIG_CHECKflag as kill-switch, per the one-flag-per-cron idiom.Blockchainenum) are skipped via try/catch, same as existinggetClientconsumers handle them.This gives the alert a continuous signal: the follow-up in the server repo shrinks the alert window to 20m over this heartbeat, so it keeps firing until the config is actually restored and resolves shortly after.
Verification
jest blockchain-config-check→ 3 passed; eslint + prettier +tsc --noEmitclean.