Enforce cooperative SQL deadlines and release flyquery 26.10.3

CHANGELOG.md

@@ -5,6 +5,34 @@ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
 the project uses [CalVer](https://calver.org/) (YY.MM.PP) per the
 Firefly Framework convention (memory: `firefly_uses_calver`).
 
+## [26.10.3] - 2026-10-14
+
+### Fixed
+
+- Enforce the positive `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` setting for
+  each DuckDB attempt. A monotonic deadline includes queueing, setup, SQL,
+  row fetching and copy-on-write export. A watchdog interrupts the native
+  connection; expired attempts cannot return a successful result or mutation.
+- Preserve task cancellation while waiting for the native worker to close.
+  Each call owns its connection. Cleanup failures remain chained to the
+  cancellation exception, and copy-on-write scratch files are removed even
+  when connection close reports an error. Original snapshots remain unchanged.
+- Record timed-out statements as `FAILED` with SQL state `57014` in query
+  history. Critic does not repair this resource failure, and a timeout during
+  repair cannot restore an earlier success as the final result.
+- Keep watchdog waits within native threading limits for large positive
+  integer budgets. Retain transient interruption failures and retry them.
+
+### Operational scope
+
+- The deadline is a cooperative, per-attempt DuckDB budget. It is not a hard
+  process kill, an operating-system sandbox, or an end-to-end request SLA.
+  Model calls, retrieval and object-store work outside the executor have
+  separate budgets. Candidate selection keeps its existing finite policy.
+- Query DTOs and the seven typed agent page APIs are unchanged. Python and
+  Java SDK metadata now report 26.10.3. The framework pins and the nineteen
+  historical source contracts plus two reviewed service variants are retained.
+
 ## [26.10.2] - 2026-10-14
 
 ### Changed

README.md

@@ -30,7 +30,7 @@ JSONL, Parquet, Avro, ORC, Arrow and Feather, plus their `.gz`,
 [![OpenAPI](https://img.shields.io/badge/api-openapi%203.1-green)](docs/api-reference.md)
 [![DuckDB](https://img.shields.io/badge/query%20engine-duckdb-yellow)](docs/architecture.md)
 [![pgvector](https://img.shields.io/badge/vector--store-pgvector-336791)](docs/architecture.md)
-[![Version](https://img.shields.io/badge/version-26.10.2-green.svg)](#)
+[![Version](https://img.shields.io/badge/version-26.10.3-green.svg)](#)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
 
 </div>
@@ -287,8 +287,8 @@ Postgres stores metadata, embeddings, relations, audit, and agent tokens.
 Cloud storage and ingestion adapters are separate from the qualified local
 query workflow. The [SQL read boundary](docs/security-model.md#sql-read-boundary)
 explains the exact file allowlist, copy-on-write, and upstream authorization.
-Statement timeouts are currently unenforced; these in-process controls do not
-constitute an operating-system SQL sandbox.
+Each execution attempt has a configurable cooperative DuckDB deadline.
+These in-process controls do not constitute an operating-system SQL sandbox.
 
 The **hexagonal adapter ring** means every external boundary (object store, file
 reader, vector store, PII scanner, semantic compiler, rate limiter) is a Python

Taskfile.yml

@@ -117,7 +117,7 @@ tasks:
           -g python
           -o sdks/python
           --package-name flyquery_sdk
-          --additional-properties=packageVersion=26.10.2,projectName=flyquery-sdk,library=asyncio
+          --additional-properties=packageVersion=26.10.3,projectName=flyquery-sdk,library=asyncio
           --skip-validate-spec
       - cd sdks/python && uv pip install --system --quiet . || true
 
@@ -142,5 +142,5 @@ tasks:
           --invoker-package com.firefly.flyquery
           --api-package com.firefly.flyquery.api
           --model-package com.firefly.flyquery.model
-          --additional-properties=artifactId=flyquery-sdk,groupId=com.firefly,artifactVersion=26.10.2,library=webclient,dateLibrary=java8,java8=true,useJakartaEe=true
+          --additional-properties=artifactId=flyquery-sdk,groupId=com.firefly,artifactVersion=26.10.3,library=webclient,dateLibrary=java8,java8=true,useJakartaEe=true
           --skip-validate-spec

docs/api-reference.md

@@ -758,8 +758,8 @@ the full pipeline completes.
 ```
 
 The current request accepts `question`, `dataset_id`, and optional
-`conversation_id`. Row caps use the executor's global setting; statement
-timeouts are unenforced. There are no per-request `row_cap` or
+`conversation_id`. Row caps and the cooperative statement deadline use the executor's global
+settings. There are no per-request `row_cap` or
 `statement_timeout_ms` overrides. See the
 [SQL read boundary](security-model.md#sql-read-boundary).
 

docs/architecture.md

@@ -223,7 +223,7 @@ flyquery uses a two-tier storage design:
    registered local Parquet snapshots in a fresh in-memory connection.
    Connections have exact file allowlists and locked external-access settings.
    See the [SQL read boundary](security-model.md#sql-read-boundary) for admission
-   rules, copy-on-write, and the unenforced statement timeout. ObjectStore
+   rules, copy-on-write, and the cooperative per-attempt statement deadline. ObjectStore
    provider support does not establish remote query support.
 
 3. **Postgres + pgvector** — metadata, schema knowledge base, relations,
@@ -484,7 +484,7 @@ DuckDB executor (in-process, per-request)
   create views over registered local Parquet; exact allowed_paths
   external access / extension autoload off; configuration locked
   SET memory_limit; LIMIT default_row_cap+1 (SELECT)
-  statement timeout is unenforced
+  positive per-attempt deadline; native interrupt + monotonic checkpoints
        │
   ┌────┴──────┐
   OK        ERROR
@@ -517,7 +517,7 @@ metadata in both synchronous and SSE handlers. See the
 |-------|-------------|-------|
 | `GroundingAgent` | `GroundedContext` | Iterative schema expansion up to `EXPAND_ITERS=2` |
 | `GenerationAgent` | `GeneratedCandidates` | `FLYQUERY_GENERATION_CANDIDATES=3`; semantic-layer trusted over uploaded tables |
-| `CriticAgent` | `RefinedSql` | Always invoked on error; optionally on success to rank N>1 |
+| `CriticAgent` | `RefinedSql` | Repairs eligible SQL errors; timeout state `57014` is terminal; may reconsider an executed result |
 | `ExplainerAgent` | `ResultExplanation` | NL summary + chart hint; cheap model by default |
 
 All agents follow canon's `build_agent` recipe: structured `output_type`,

docs/concurrency.md

@@ -215,8 +215,10 @@ Properties of this design:
 
 The connection is not read-only. SQL admission and snapshot checks precede
 execution; see the [SQL read boundary](security-model.md#sql-read-boundary)
-for the full policy. The global statement-timeout setting is unenforced,
-so the SELECT row cap does not bound the time required to compute rows.
+for the full policy. Each call has its own positive cooperative deadline
+and watchdog, which interrupts only that call's native connection. Cancellation
+waits for worker cleanup. The SELECT row cap limits output size, not compute time;
+use deployment limits for hard CPU/memory isolation.
 
 **Concurrency concern:** DuckDB in-process with asyncio requires care. DuckDB
 operations are synchronous and CPU-bound. They block the asyncio event loop

docs/deployment.md

@@ -140,16 +140,18 @@ is the canonical reference. Copy to `.env` and edit.
 |----------|---------|-------|
 | `FLYQUERY_DUCKDB_MEMORY_LIMIT` | `4GB` | Per-connection memory cap |
 | `FLYQUERY_DEFAULT_ROW_CAP` | `1000` | Default result row limit |
-| `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` | `30000` | Configuration field; currently unenforced by DuckDBExecutor |
+| `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` | `30000` | Positive per-attempt cooperative native deadline; not a hard OS budget |
 | `FLYQUERY_RESULT_TTL_HOURS` | `24` | Presigned URL and result object TTL |
 | `FLYQUERY_RESULT_PREVIEW_MAX_BYTES` | `131072` | Preview JSON size cap (128 KiB) |
 | `FLYQUERY_DUCKDB_HTTPFS` | `true` | Configuration field; does not enable httpfs in DuckDBExecutor |
 
 The qualified query workflow uses registered local Parquet with DuckDB >= 1.3.2.
 External access and extension autoload/install remain disabled; see the
 [SQL read boundary](security-model.md#sql-read-boundary). Query requests have
-no per-request row-cap or statement-timeout override, and the global timeout
-setting does not stop a long-running statement.
+no per-request row-cap or statement-timeout override. The positive global
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` setting (30,000 ms by default) applies
+a cooperative native deadline to each executor attempt. Continue to configure
+process/container resource limits; this does not replace an OS-level sandbox.
 
 ### Ingestion
 

docs/operations-runbook.md

@@ -255,10 +255,13 @@ the `TenantContextMiddleware` log output.
 failure, inspect rate-limit and latency errors from the configured provider.
 For DuckDB, inspect the query plan and reduce expensive scans or joins.
 
-`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` is currently unenforced by
-DuckDBExecutor, and query requests have no timeout override. Raising it
-does not stop a long-running SQL statement. The SELECT row cap limits
-returned rows, not computation time; see the
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` controls a positive cooperative
+per-attempt deadline (30,000 ms by default). Expiry interrupts that native
+connection and records `FAILED` with SQL state `57014`; Critic does not
+repair this failure. Raising the setting permits more execution time, and
+query requests have no timeout override. Keep process/container limits for
+uncooperative native or filesystem work. The SELECT row cap limits returned
+rows, not computation time; see the
 [SQL read boundary](security-model.md#sql-read-boundary).
 
 ---

docs/payload-reference.md

@@ -303,8 +303,8 @@ Sent to `PUT /api/v1/schema-objects/{id}`:
 | `conversation_id` | `string \| null` | No | For multi-turn drill-down |
 
 The current DTO has no `row_cap` or `statement_timeout_ms` fields. The
-executor uses `FLYQUERY_DEFAULT_ROW_CAP`; statement timeouts are currently
-unenforced. See the [SQL read boundary](security-model.md#sql-read-boundary).
+executor uses `FLYQUERY_DEFAULT_ROW_CAP` and the positive, per-attempt
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` cooperative deadline. See the [SQL read boundary](security-model.md#sql-read-boundary).
 
 ### AnswerResponse
 

docs/pipeline.md

@@ -291,13 +291,14 @@ DuckDB executor (in-process, per-request)
   Disable external access and extension autoload/install; lock configuration
   SET memory_limit = FLYQUERY_DUCKDB_MEMORY_LIMIT (default 4GB)
   LIMIT default_row_cap+1 to detect SELECT overflow
-  Statement timeout is unenforced; connection is not read-only
+  Per-attempt cooperative deadline interrupts DuckDB; connection is not read-only
 
         │
    ┌────┴──────┐
    OK         ERROR
    │           │
-   │          CriticAgent (up to FLYQUERY_MAX_REFINE_RETRIES=2)
+   │          SQL state 57014 -> FAILED audit; no Critic
+   │          Other errors -> CriticAgent (up to FLYQUERY_MAX_REFINE_RETRIES=2)
    │           │  output: RefinedSql {sql, reasoning, confidence}
    │           │  loop back to AST check
    │           │
@@ -353,7 +354,7 @@ Parquet workflow, copy-on-write, and remaining limits.
 |-------|-------------|---------------|-------|
 | `GroundingAgent` | `GroundedContext` | `FLYQUERY_GROUNDING_MODEL` (claude-sonnet-4-6) | Iterative expansion up to `EXPAND_ITERS=2` |
 | `GenerationAgent` | `GeneratedCandidates` | `FLYQUERY_GENERATION_MODEL` (claude-sonnet-4-6) | N candidates (default 3); delta SQL when conversation active |
-| `CriticAgent` | `RefinedSql` | `FLYQUERY_CRITIC_MODEL` (claude-sonnet-4-6) | Always invoked on error; optionally on success to rank N>1 |
+| `CriticAgent` | `RefinedSql` | `FLYQUERY_CRITIC_MODEL` (claude-sonnet-4-6) | Repairs eligible SQL errors; timeout state `57014` is terminal; may reconsider an executed result |
 | `ExplainerAgent` | `ResultExplanation` | `FLYQUERY_EXPLAINER_MODEL` (claude-haiku-4-5) | Cheap model; 50-row preview input |
 
 ### SSE frames on /query/stream

docs/sdks.md

@@ -21,10 +21,10 @@ Install the Python wheel from the canonical GitHub Release, rather than assuming
 that a package with the same name on PyPI is maintained by this project:
 
 ```bash
-uv add https://github.com/fireflyframework/flyquery/releases/download/v26.10.2/flyquery_sdk-26.10.2-py3-none-any.whl
+uv add https://github.com/fireflyframework/flyquery/releases/download/v26.10.3/flyquery_sdk-26.10.3-py3-none-any.whl
 ```
 
-Keep service and client versions aligned. This release uses version `26.10.2`;
+Keep service and client versions aligned. This release uses version `26.10.3`;
 read the release notes and API snapshot before upgrading. The repository also
 retains earlier releases and their original assets.
 

docs/security-model.md

@@ -262,10 +262,35 @@ upstream responsibilities. Local handler tests with repository doubles do
 not establish PostgreSQL RLS acceptance. The in-process settings provide
 defense in depth; they do not constitute an operating-system SQL sandbox.
 
-Statement timeouts are currently unenforced. `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS`
-does not interrupt DuckDB execution, and the current `QueryRequest` has no
-`statement_timeout_ms` or `row_cap` override. Result limits do not bound CPU
-time spent scanning, joining, or aggregating before rows are returned.
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` sets a positive, per-attempt deadline
+(default: 30,000 ms). It starts when the executor is called and includes the
+worker queue, connection setup, SQL, fetching rows, and copy-on-write export.
+At expiry, a watchdog calls DuckDB's native `interrupt()` on that attempt's
+connection. Monotonic checks prevent a late result or mutation from being
+returned. The executor closes the connection and removes its scratch output
+before completing. Original snapshot files remain unchanged.
+
+A timeout returns `ExecutionError` with SQL state `57014`. Direct SQL and
+SSE handlers record `FAILED` and the state in query history. The query
+orchestrator does not ask the Critic to repair a timed-out statement, and a
+timeout during repair cannot restore an earlier success as the final result.
+Candidate selection may still run other candidates under the existing finite
+candidate policy; each attempt receives its own deadline.
+
+Cancelling the asynchronous executor call interrupts only its own native
+connection, waits for that worker's cleanup, and propagates cancellation.
+Other calls use separate connections. Unexpected cleanup failures remain
+chained to the cancellation exception. If a native interruption fails, the
+watchdog retries and the timeout error retains the exception type.
+Cancelling a caller does not guarantee
+that an HTTP disconnect is forwarded by every server or streaming adapter.
+
+This is a cooperative DuckDB interruption, not a hard operating-system kill
+or an end-to-end request SLA. Native work or filesystem calls that do not
+cooperate can delay cleanup. Model generation, retrieval, object-store
+transfers, and snapshot publication outside the executor have separate
+budgets. The current `QueryRequest` has no `statement_timeout_ms` or `row_cap`
+override. Result limits alone do not bound scanning, joins, or aggregation.
 
 ObjectStore upload, ingestion, and result-storage adapters are separate from
 this query boundary. Remote object-store and extension-backed query workflows
@@ -372,7 +397,7 @@ Files exceeding the per-file cap are rejected at the upload endpoint with
 | Zip bomb | Decompressed-size check during decompression; capped at 2× the compressed size or `FLYQUERY_MAX_FILE_MB`, whichever is smaller |
 | ZIP with multiple entries | Rejected immediately with `error_code=ZIP_MULTI_ENTRY` |
 | Polyglot file (valid magic bytes for two formats) | Stage 1 verifies magic bytes match declared extension; mismatch → `FORMAT_MISMATCH` |
-| Billion-laughs JSON | Apply deployment resource limits to ingestion; DuckDB query statement timeout is unenforced (see [SQL read boundary](#sql-read-boundary)) |
+| Billion-laughs JSON | Apply deployment resource limits to ingestion; use the cooperative DuckDB deadline alongside deployment resource limits (see [SQL read boundary](#sql-read-boundary)) |
 
 ### GDPR purge
 

docs/troubleshooting.md

@@ -212,10 +212,13 @@ for the test role setup recipe.
 
 ### Symptom: Query fails with `execution_status=FAILED`, error contains "timeout"
 
-`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` is currently unenforced by the query
-executor, and `QueryRequest` has no per-query timeout override. Increasing
-that setting cannot fix a DuckDB execution stall. Identify the failing stage
-from the error and logs; see the [SQL read boundary](security-model.md#sql-read-boundary).
+A DuckDB deadline appears as `FAILED` with SQL state `57014` in query history.
+Inspect the plan before raising the positive per-attempt
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` setting; a higher value permits more
+execution time. It does not change provider/model timeouts or guarantee a hard
+process kill. `QueryRequest` has no per-query timeout override. Identify the
+failing stage from the error and logs; see the
+[SQL read boundary](security-model.md#sql-read-boundary).
 
 **Root cause A:** LLM API latency (Anthropic / OpenAI rate limit or slow
 response).
@@ -294,8 +297,11 @@ For repeated OOM on the same query, simplify the question or add a WHERE
 clause to scope the data.
 
 For a long-running DuckDB statement, reduce the scan or join work and inspect
-the query plan. The statement-timeout setting currently does not interrupt
-execution; the result row cap limits returned rows, not computation time.
+the query plan. `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` interrupts the native
+connection when an attempt expires. Query history records `FAILED` and SQL
+state `57014`; Critic does not repair this resource failure. Each candidate
+attempt has its own budget. The result row cap limits returned rows, not
+computation time, and cooperative interruption cannot replace process limits.
 File-source or unregistered-table rejections are expected policy failures,
 not reasons to enable httpfs or external access. See the
 [SQL read boundary](security-model.md#sql-read-boundary).

openapi.json

@@ -6206,7 +6206,7 @@
       "url": "https://github.com/fireflyframework/flyquery/blob/main/LICENSE"
     },
     "title": "flyquery",
-    "version": "26.10.2"
+    "version": "26.10.3"
   },
   "openapi": "3.1.0",
   "paths": {

pyfly.yaml

@@ -12,7 +12,7 @@ pyfly:
   # and admin tooling identify the service correctly.
   app:
     name: flyquery
-    version: 26.10.2
+    version: 26.10.3
     description: >-
       Operational Structured-Data Intelligence (upload-driven).
       Multi-tenant ingestion + Text-to-SQL over user-uploaded structured

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "flyquery"
 # CalVer YY.MM.PP per memory `firefly_uses_calver`.
-version = "26.10.2"
+version = "26.10.3"
 description = "Operational Structured-Data Intelligence (upload-driven) -- multi-tenant ingestion + Text-to-SQL service over user-uploaded structured files. Part of Firefly Framework."
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

sdks/java/README.md

@@ -18,19 +18,19 @@ default headers.
 
 ## Install
 
-For release 26.10.2, use the matching Maven coordinate:
+For release 26.10.3, use the matching Maven coordinate:
 
 ```xml
 <dependency>
   <groupId>com.firefly</groupId>
   <artifactId>flyquery-sdk</artifactId>
-  <version>26.10.2</version>
+  <version>26.10.3</version>
 </dependency>
 ```
 
 The publisher targets GitHub Packages at
 `https://maven.pkg.github.com/fireflyframework/flyquery`. GitHub Packages requires
-Maven authentication. See the [release](https://github.com/fireflyframework/flyquery/releases/tag/v26.10.2)
+Maven authentication. See the [release](https://github.com/fireflyframework/flyquery/releases/tag/v26.10.3)
 for downloadable artifacts and use the service's matching version. Earlier
 release artifacts keep their original registry coordinates.
 

sdks/java/api/openapi.yaml

@@ -13,7 +13,7 @@ info:
     name: Apache-2.0
     url: https://github.com/fireflyframework/flyquery/blob/main/LICENSE
   title: flyquery
-  version: 26.10.2
+  version: 26.10.3
 servers:
 - description: This service
   url: /

sdks/java/build.gradle

@@ -2,7 +2,7 @@ apply plugin: 'idea'
 apply plugin: 'eclipse'
 
 group = 'com.firefly'
-version = '26.10.2'
+version = '26.10.3'
 
 buildscript {
     repositories {