Conversation
A queued row that failed a drain attempt was terminal. The failure reason muted it in every automatic query — the idle drain, the due sweep, and every wait-keyed wake all filter on `failure_reason IS NULL`, as do the claim gate and the group-eligibility predicate — and the only writers that clear it need a claim those same queries refuse to hand out. Nothing automatic could reach the row again, and the UI offers no send on a row whose wait is one of the kinds it will not send now, so the message needed a `bb thread queue send` to move at all. That would be defensible if a failure said something durable about the message. It does not: it says what the server managed at one instant, and the instant that fails a dispatch is usually a restart. A provider whose plugin is still loading answers `provider_bridge_unavailable` — "may be disabled or still building" — and the row it strands is one queued before the restart it was waiting out. A failure now spends one of the row's attempts and books the next on a widening ladder: `failure_count` and `next_attempt_at` beside the reason, five attempts across about an hour, `next_attempt_at` NULL recording a budget that ran out. The column is deliberately not named `retry_*`: this table's `retry_of_turn_request_id`, `retry_attempt` and `retry_reason` all mean the re-submission of a turn that ran and failed at the provider, which is a different thing from a row that never got out of the queue. A new `failed-retry` wake on the periodic sweep re-attempts the rows that have come due, and it is the one automatic claim that may see a recorded failure. It re-asks the whole dispatch question rather than waiting on the wait's own edge, because a row that sat failed may have missed that edge entirely — a `host-offline` row whose host came back while it waited is exactly the shape that got stuck. A fresh, successful statement of the row's wait resets the budget, since the attempt that wrote it knows something newer than the failure did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ymichael
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human comments
What was wrong
A failed queued-message dispatch recorded a failure reason that excluded the message from every automatic drain and claim path. Temporary startup failures could strand a message after the condition cleared. Failed rows with certain waits also hid the app's Send now button, while CLI queue listings showed only the old wait instead of the failure.
What changed
Persist a failure count and next-attempt time and retry eligible failures automatically after delays of 15 seconds, 1 minute, and 5 minutes: three retries after the initial attempt. Ordinary drains continue skipping failed rows. The retry sweep checks current eligibility, including hosts that already reconnected. A fresh wait or successful requeue resets the failure budget; exhausted messages remain failed for manual handling.
Failed rows expose the existing Send now button with its existing behavior. CLI queue listings show the failure instead of the previous wait, print the full failure reason, and provide the exact
bb thread queue send <thread-id> <message-id>recovery command. The CLI guide and skill document manual recovery and the fact that editing does not trigger it.The migration adds two database columns. Existing failed messages are not backfilled with scheduled retries, but can use manual recovery. No new public API or server/daemon wire fields.
How you verified
pnpm exec turbo run typecheck --filter=@bb/app --filter=@bb/cli --filter=@bb/server— passed.pnpm exec turbo run test --filter=@bb/app --filter=@bb/cli --filter=@bb/server -- queued-message-wait QueuedMessagesList thread-organization queue-drain-failure— passed: 83 app, 9 CLI, and 12 server tests. UI tests click Send now on failed rows for all five previously hidden wait types; CLI tests verify full failures and recovery commands for scoped and cross-thread lists.