Skip to content

fix(payout): classify Monero daemon-connection failures as pre-broadcast so transient wallet-rpc wedges self-heal#4342

Draft
davidleomay wants to merge 1 commit into
developfrom
fix/monero-payout-daemon-connection-classifier
Draft

fix(payout): classify Monero daemon-connection failures as pre-broadcast so transient wallet-rpc wedges self-heal#4342
davidleomay wants to merge 1 commit into
developfrom
fix/monero-payout-daemon-connection-classifier

Conversation

@davidleomay

Copy link
Copy Markdown
Member

Summary

Two Monero payout incidents both escalated to PayoutUncertain in a state where the wallet had provably not signed or broadcast anything — monero-wallet-rpc's persistent daemon socket was wedged and returned two different pre-signing errors that the classifier introduced in #4238 wasn't allowlisting:

  • 2026-07-18 06:03 UTC — order 112451: code -4, "failed to get output distribution"
  • 2026-07-20 16:36 UTC — order 112677: code -38, "no connection to daemon"

Both classes are constructed inside wallet2::transfer_selected_rct before ring signatures; both are self-recoverable transient wedges (the wallet reconnected on its own within minutes). With the fail-closed broadcast boundary, however, every send-RPC failure without an allowlisted pre-broadcast marker escalates to PayoutUncertain in ~30s — so both incidents needed manual reset even though the API's pre-broadcast retry loop would have absorbed them.

This PR routes both classes into the existing maxPreBroadcastRetries loop:

  • -38 joins MONERO_PRE_BROADCAST_RPC_CODES directly. Cited from src/wallet/wallet_rpc_server_error_codes.h; mapped in wallet_rpc_server::handle_rpc_exception before signing (source: src/wallet/wallet_rpc_server.cpp).
  • (-4, "failed to get output distribution") uses a new narrow (code, message) allowlist because -4 is GENERIC_TRANSFER_ERROR (Monero's catch-all — cannot allowlist by code alone). The what() message is hard-coded in struct get_output_distribution (src/wallet/wallet_errors.h) and only thrown during decoy selection (src/wallet/wallet2.cpp).

Message matching is exact (not substring/regex): a rewording upstream should fail closed on the next release rather than silently expand the allowlist. Bitcoin/Firo/Zano keep the unchanged 2-arg call via a default empty allowlist parameter.

With maxPreBroadcastRetries = 3 (default), a transient wedge is now absorbed within ~90 seconds, matching the observed self-recovery timing on both incidents.

Test plan

  • Extended classifier spec: 3 new tests (-38 allowlist, exact-message match, wrong-code and substring-match negatives)
  • Extended monero-client spec: 2 new tests (-38 and (code=-4, message="failed to get output distribution") allowlist entries)
  • Full suite: jest tx-broadcast.error|monero-client|payout-bitcoin-based → 85/85 pass
  • tsc --noEmit clean (Bitcoin/Firo/Zano callers keep the 2-arg call via default empty allowlist)
  • eslint clean on all 4 changed files
  • Manual verification (this branch → dev → observe next wedge event self-heal)

Related

…ast so transient wallet-rpc wedges self-heal

Two Monero payout incidents (2026-07-18 order 112451, 2026-07-20 order 112677)
both escalated to PayoutUncertain in a state where the wallet had provably not
signed or broadcast anything: monero-wallet-rpc reported code -38
"no connection to daemon" and code -4 "failed to get output distribution" during
the `transfer` call while its persistent daemon socket was wedged. Since the
fail-closed broadcast boundary landed in #4238, every send-RPC failure without
an allowlisted pre-broadcast marker escalates in ~30s — and neither of these
markers was on the list, even though both classes are constructed inside
wallet2 before ring signatures.

Both codes now stay plain and route into the existing pre-broadcast retry loop:
- WALLET_RPC_ERROR_CODE_NO_DAEMON_CONNECTION (-38): mapped in
  wallet_rpc_server::handle_rpc_exception before signing, cited from
  src/wallet/wallet_rpc_server_error_codes.h.
- (-4, "failed to get output distribution"): fall-through to
  GENERIC_TRANSFER_ERROR because handle_rpc_exception has no dedicated catch
  for error::get_output_distribution; the exact what() message is hard-coded
  in src/wallet/wallet_errors.h and thrown only during decoy selection
  (src/wallet/wallet2.cpp).

The classifier gains an optional (code, message) allowlist parameter with
exact-string matching only — a substring/regex match would erode the guarantee
that only the specific pre-signing wallet exception surfaces, and a rewording
upstream should fail closed on the next release rather than silently expand
the allowlist. Bitcoin/Firo/Zano keep the unchanged 2-arg call via a default
empty allowlist.

With the default maxPreBroadcastRetries=3, a transient wedge is now absorbed
within ~90 seconds instead of escalating, matching the recovery pattern
observed on both incidents.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant