Investigate DB connection drop causing TRILOGY_CLOSED_CONNECTION 500s

[maebeale]

Summary

Production threw ActiveRecord::ConnectionFailed: Trilogy::EOFError: trilogy_query_recv: TRILOGY_CLOSED_CONNECTION on GET / (home#index). The immediate symptom — a 500 on the public homepage — was patched in #2117 by making the decorative banner query degrade gracefully. This issue tracks the underlying cause: the DB connection itself was dropped mid-request.

The banner was just the first query on the page, so it was the messenger, not the culprit. Any request whose first DB query lands on a stale/dropped connection would have 500'd the same way.

What happened

• Error: ActiveRecord::ConnectionFailed: Trilogy::EOFError: trilogy_query_recv: TRILOGY_CLOSED_CONNECTION
• Where: ApplicationHelper#display_banner (app/helpers/application_helper.rb) → Banner.published — the first query rendered on the homepage layout
• When: 2026-08-09 ~01:09 AM ET (2026-08-09 05:09:29 UTC)
• Request: GET https://portal.awbw.org/, from a DigitalOcean Uptime Probe
• Host: awbw-production-6d475b8bcc-7drhj (production, puma 7.2.1, pid 1)
• Honeybadger request id: 0ca65b06-f379-4f9b-b884-126051804d96
• Frequency so far: 1 occurrence

TRILOGY_CLOSED_CONNECTION / EOFError on trilogy_query_recv means the server closed the TCP connection and Rails then tried to run a query over that dead socket. This is a classic stale pooled connection: a connection sits idle in the pool longer than the server (or an intermediary) is willing to keep it, the far end closes it, and the next checkout reuses it without noticing it's dead.

Likely causes (to investigate)

Common reasons an idle pooled connection goes dead:

1. MySQL wait_timeout / interactive_timeout on the DB server closing idle connections that Rails still holds in its pool.
2. A proxy / load balancer idle timeout between the app and the database (e.g. a managed-DB connection pooler, k8s service, or DO managed DB idle cutoff) severing long-idle connections.
3. A DB server restart / failover / maintenance at ~01:09 ET that dropped all live connections.
4. No connection liveness/reaping config — see below; Rails isn't set up to proactively verify or recycle idle connections.

Current config (relevant gaps)

config/database.yml base anchor:

• adapter: trilogy, pool: ENV["DB_POOL"] || 10
• No checkout_timeout, reaping_frequency, idle_timeout, connect_timeout, or read_timeout set.

config/puma.rb: threads = RAILS_MAX_THREADS || 3 (so the default pool of 10 comfortably covers threads — pool exhaustion is not the issue here).

Rails/Trilogy do verify a connection on checkout in some paths, but there's no reaping_frequency/idle_timeout to proactively prune connections that have been idle long enough to be reaped by the server, and no explicit timeouts to bound a hung socket.

Proposed investigation / fixes

• Check the production MySQL wait_timeout / interactive_timeout values and compare against how long connections sit idle in the pool.
• Confirm whether there's a proxy/pooler (or DO managed-DB) idle timeout between app and DB, and its value.
• Check DB server logs around 2026-08-09 05:09 UTC for a restart / failover / connection reset.
• Consider setting reaping_frequency and/or idle_timeout in database.yml so stale connections are pruned before they're handed to a request.
• Consider explicit connect_timeout / read_timeout so a dead socket fails fast and predictably.
• Decide whether transient connection failures on read paths should be retried once (framework-level) rather than surfaced as 500s.

Notes

• Don't 500 the homepage when a DB blip breaks the banner query #2117 only prevents the banner from 500ing the page; it does not address the connection drop. If this recurs on other actions, that PR won't help them.
• If this stays at 1 occurrence and correlates with a one-off DB restart/failover, this may be benign — but it's worth confirming rather than assuming.