Skip to content

feat(blockchain): log unconfigured clients periodically for persistent alerting#4235

Open
Danswar wants to merge 2 commits into
developfrom
feat/unconfigured-client-heartbeat
Open

feat(blockchain): log unconfigured clients periodically for persistent alerting#4235
Danswar wants to merge 2 commits into
developfrom
feat/unconfigured-client-heartbeat

Conversation

@Danswar

@Danswar Danswar commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Why

#3878 (Cardano) and #4203 (Solana) deliberately downgrade an unset Tatum API key from a recurring ERROR to a warn-once-per-boot skip, because dfxdev runs keyless by design. DFXServer/server#730 turned that boot warn into a prod alert, but since the warn is logged only once per boot, the alert's 24h Loki window auto-resolves even if the key is still missing — a resolve doesn't mean the config was restored (DFXServer/server#753).

What

New BlockchainConfigCheckService (blockchain shared services):

  • Every 5 minutes, sweeps BlockchainRegistryService for clients reporting isConfigured === false and logs one warn line: Blockchain clients not configured: <chains> — chain names only, no secret values.
  • Gated to prd: on dev/loc an unconfigured Tatum client is the intended state, and a periodic warn there would be permanent noise.
  • Own Process.BLOCKCHAIN_CONFIG_CHECK flag as kill-switch, per the one-flag-per-cron idiom.
  • Chains without a registered service (exchanges, banks, payment providers in the Blockchain enum) are skipped via try/catch, same as existing getClient consumers handle them.

This gives the alert a continuous signal: the follow-up in the server repo shrinks the alert window to 20m over this heartbeat, so it keeps firing until the config is actually restored and resolves shortly after.

Verification

  • New spec covers: warn repeats on every tick (the persistent property), lists exactly the unconfigured chains, silent when all configured, silent outside prd.
  • jest blockchain-config-check → 3 passed; eslint + prettier + tsc --noEmit clean.

@github-actions

Copy link
Copy Markdown

⚠️ Unverified Commits (1)

The following commits are not signed/verified:

  • f0d3367 feat(blockchain): log unconfigured clients periodically for persistent alerting (Daniel Padrino)
How to sign commits
# SSH signing (recommended)
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true

# Re-sign last commit
git commit --amend -S --no-edit
git push --force-with-lease

❌ Security: 1 critical vulnerabilities

@Danswar
Danswar force-pushed the feat/unconfigured-client-heartbeat branch 2 times, most recently from b6ea735 to 08f81c7 Compare July 16, 2026 14:48
…t alerting

api#3878 and api#4203 downgraded an unset Tatum API key to a warn-once-
per-boot skip, so a lost prod secret only leaves a single boot-time log
line and the infra alert auto-resolves after its lookback window even if
the key is still missing (DFXServer/server#753).

BlockchainConfigCheckService now re-logs the unconfigured state every
5 minutes on prd (chain names only, no secret values), giving the alert
a continuous signal that stops only when the config is restored. Gated
to prd since unconfigured clients are the intended state elsewhere, with
an own Process flag as kill-switch.
@Danswar
Danswar force-pushed the feat/unconfigured-client-heartbeat branch from 08f81c7 to d8e039d Compare July 16, 2026 18:17
@TaprootFreak

Copy link
Copy Markdown
Collaborator

Reviewed. The heartbeat itself is fine — 5-min interval with jitter, prd-only guard with an early return before any registry call, the DisabledProcess kill-switch is wired, the warn level matches the "not configured" convention elsewhere, and configured clients are only read (no side effects). The 3 tests pass and tsc is clean.

One thing I'd fix before merging, though: the blanket catch doesn't only absorb the intended "No service found for blockchain" enum-mismatch — it also swallows a real, present-day TypeError. registryService.getClient(chain) can return a falsy client today (e.g. BitcoinService.getDefaultClient() returns null when the node URL env var is unset; FiroService.client is undefined without FIRO_NODE_URL), so .getClient(chain).isConfigured throws Cannot read properties of null, which the catch treats identically to the enum case and quietly filters the chain out of the alert — i.e. exactly the config problem this service exists to surface, silently absorbed.

Two sibling services already handle this explicitly: log-job.service.ts:346 and payment-balance.service.ts:81 both do if (!client) warn(...). It's harmless in practice today only because those chains never override the default isConfigured getter, but that's incidental. Suggest narrowing the catch to the specific "No service found" message, or checking !client first before touching .isConfigured.

The periodic config check caught every error, including a real TypeError when
getClient returns a null/undefined client (e.g. an unset node URL for
Bitcoin/Firo) — silently dropping exactly the chain it should surface. Check
for a falsy client and warn (matching log-job/payment-balance) before reading
isConfigured, and narrow the catch to the "No service found" case so genuine
errors stay visible.
@TaprootFreak

Copy link
Copy Markdown
Collaborator

Pushed 670aa1a for the swallowed-error point: it now checks for a falsy client and warns (matching log-job/payment-balance) before touching .isConfigured, and narrows the catch to the "No service found" case so genuine errors stay visible. blockchain-config-check jest + tsc green locally.

@TaprootFreak

Copy link
Copy Markdown
Collaborator

A deeper full-PR pass surfaced a few coverage limitations worth your call — these are about which chains the sweep can actually see (feature scope), not bugs in the heartbeat itself:

  • Lightning (and any coin-only chain) is silently excluded: BlockchainRegistryService.getService() has no case for Blockchain.LIGHTNING, so getClient(LIGHTNING) throws "No service found", which the catch treats as "configured, skip".
  • Bitcoin's payout-critical output client is never inspected: getClient(chain) = getService(chain).getDefaultClient() with no type argument, so the OUT client isn't part of the sweep.
  • Monero, Zano, Spark, Arkade can never be reported as unconfigured: their services construct a client unconditionally with no null-guard, so a missing config there is invisible to the sweep under any state.
  • BitcoinTestnet4's default client can be null when NODE_BTC_TESTNET4_OUT_URL_ACTIVE is unset → it would flag as unconfigured every tick; worth confirming that var is set in PRD.
  • Nit (CONTRIBUTING): the new file uses relative imports (../enums/…, ./blockchain-registry.service) — CONTRIBUTING asks for absolute src/… paths.
  • Test gap: the new spec mocks BlockchainRegistryService entirely, so none of the above topology is exercised.

The null-client warn fix (670aa1a) is solid for the chains the sweep does reach. Whether to widen coverage or just document the intended scope with a short code comment is your design call — happy to implement whichever you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants