Skip to content

[fix](arrow-flight) Bound idle Arrow Flight SQL sessions separately from wait_timeout - #67504

Open
raghav-reglobe wants to merge 1 commit into
apache:masterfrom
raghav-reglobe:fix-arrow-flight-session-idle-timeout
Open

[fix](arrow-flight) Bound idle Arrow Flight SQL sessions separately from wait_timeout#67504
raghav-reglobe wants to merge 1 commit into
apache:masterfrom
raghav-reglobe:fix-arrow-flight-session-idle-timeout

Conversation

@raghav-reglobe

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67503

Related PR: #64799 (kept the coordinator alive across GetFlightInfo → DoGet so the BE can fetch external-table splits; this PR bounds how long an abandoned session can hold it)

Problem Summary:

A Flight SQL session idles under the same wait_timeout as a MySQL connection (28800s by default). Since #64799 the coordinator of a BE-served Flight query is kept alive until the session's next query or its close, and that coordinator holds the query's workload-group queue slot and active_queries registration. Most Flight clients open a session per query and never send CloseSession (the ADBC/JDBC drivers don't do it on connection close, as the comment in DorisFlightSqlProducer.closeSession already notes), so each abandoned session pins one slot for up to eight hours.

Observed on a 2-FE cluster with a max_concurrency=8 workload group: eight finished Flight queries sat in information_schema.active_queries as RUNNING for over an hour, and every later query in the group failed with query queue timeout, timeout: 60000 ms. Killing the idle sessions (SHOW PROCESSLIST rows with Host = 0.0.0.0:0) released the slots immediately. Full write-up in #67503.

This PR adds a mutable FE config, arrow_flight_session_idle_timeout_second (default 3600), and applies it in the existing connection timeout checker to ARROW_FLIGHT_SQL contexts only:

idle bound = min(wait_timeout, max(config, exec timeout))

The exec-timeout floor is deliberate: a Flight session is COM_SLEEP while the client drains the result via DoGet, and its idle clock runs from the query's start, so a bound below query_timeout would kill a long result stream before the query's own timeout could. 0 disables the bound. MySQL-protocol connections are unchanged; the kill log line now reports the effective idle timeout.

Verified on a test cluster with the bound set to 8s: an abandoned Flight session and its active_queries row were gone within 14s and the kill was logged as connection type: ARROW_FLIGHT_SQL, ... idle timeout: 8; an idle MySQL connection survived the same window; 0 left the Flight session alive.

Release note

Arrow Flight SQL sessions get their own idle timeout (arrow_flight_session_idle_timeout_second, default 1h), so an abandoned Flight session no longer holds its query's workload-group queue slot until wait_timeout.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test (FlightSqlSessionIdleTimeoutTest: the bound tightens but never widens wait_timeout, the exec-timeout floor, 0 disables, MySQL context untouched)
    • Manual test (steps above)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Idle Flight SQL sessions are now killed after 1h by default instead of after wait_timeout (8h); set arrow_flight_session_idle_timeout_second=0 for the previous behavior.
  • Does this need documentation?

    • No.
    • Yes. New FE config arrow_flight_session_idle_timeout_second — doris-website PR to follow once this is reviewed.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…rom wait_timeout

A Flight SQL session idles under the same wait_timeout as a MySQL
connection (28800s by default). Since the coordinator of a BE-served
Flight query is kept alive across GetFlightInfo -> DoGet until the
session's next query or its close, an abandoned session - a client that
opens a session per query and never sends CloseSession - keeps that
query's workload-group queue slot for the whole wait_timeout. Eight such
sessions fill a max_concurrency=8 group and every later query in it fails
with "query queue timeout".

Add a mutable FE config, arrow_flight_session_idle_timeout_second
(default 3600), applied by the existing connection timeout checker to
ARROW_FLIGHT_SQL contexts only as
min(wait_timeout, max(config, exec timeout)). The exec-timeout floor
matters: a Flight session is COM_SLEEP while the client drains the result
via DoGet and its idle clock runs from the query's start, so a bound
below query_timeout would kill a long result stream before the query's
own timeout could. 0 disables the bound. MySQL-protocol connections are
unchanged. The kill log line now reports the effective idle timeout.

Signed-off-by: Raghvendra Singh <raghav@cashify.in>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Arrow Flight SQL: an abandoned session holds its query's workload-group queue slot until wait_timeout (8h)

2 participants