Skip to content

perf(gs): queue sheet exports, cap unbounded queries and monitor event-loop delay#4346

Open
Danswar wants to merge 4 commits into
developfrom
feat/gs-export-queue
Open

perf(gs): queue sheet exports, cap unbounded queries and monitor event-loop delay#4346
Danswar wants to merge 4 commits into
developfrom
feat/gs-export-queue

Conversation

@Danswar

@Danswar Danswar commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Part of #4227. Investigation of the reported internal-tooling slowness (07-21/22) showed that the dominant share of slow /gs/db requests is not DB work: queries returning 0 rows from tiny tables (e.g. country, 250 rows) took 5–7.5s because sheet-export syncs fire in aligned bursts and their result post-processing runs synchronously on the event loop. Every in-flight request — including interactive support/compliance calls — absorbs those stalls. Additionally, ~22% of slow exports set no maxLine at all (the DTO leaves it optional), so unbounded full-table pulls are possible by omission.

Changes

  1. Export concurrency queue (GsService): POST /gs/db and POST /gs/db/custom now run through a QueueHandler with maxWorkParallel = 2, a 240s queue timeout (below the Apps-Script 6-min execution cap) and a 240s item timeout (a query that never settles frees its worker slot instead of wedging the queue). Exports are latency-tolerant batch consumers; interactive requests no longer compete with a whole burst at once. Both endpoints report failures (incl. queue timeouts) consistently as HTTP 400 — /gs/db/custom previously surfaced errors as opaque 500s.
  2. QueueHandler load-shedding (shared util): items whose queue timeout fired while still waiting are discarded instead of executed — previously their action ran anyway and the result was silently dropped, doubling load exactly when the queue is backed up. Covered by a new queue-handler.spec.ts.
  3. maxLine default cap (GsService): requests without maxLine (absent or null) now run with a cap of 10000 rows instead of unlimited (on the custom bank_tx export the cap applies per sub-query, and the truncation signal is derived per sub-query). Explicit values are untouched (the escape hatch for deliberate large exports, which also keeps the existing >100k alert mail reachable), and the largest observed no-limit export returns ~1.4k rows, so no existing consumer changes behavior. When a default-capped export actually hits the cap, a WARN log names the identifier and table — truncation is visible, not silent.
  4. Event-loop delay monitor (MonitorEventLoopService): logs mean / p95 / max event-loop delay every 10s (same cadence/level as the existing connection-pool monitor, gated by a new Process.MONITOR_EVENT_LOOP). Makes the next slowness report attributable in one log query instead of an elimination hunt.

Tests

  • gs.service.spec.ts: queue caps concurrency at 2 and preserves per-call results; custom exports route through the same queue; default cap applies for absent and null maxLine, explicit values pass through; queue loops stopped in teardown.
  • queue-handler.spec.ts (new): normal execution, discarded-after-timeout items never run, item timeout frees a wedged slot.
  • Changed suites: 350/350 passing; full suite at HEAD: 4775 passed / 146 skipped, 0 failures.

Verification after deploy

  • EventLoop delay lines appear in the API log; during sheet-sync bursts p95 shows the stall magnitude.
  • Interactive endpoint latency (access-log ms values) during burst minutes should drop; /gs/db throughput is unchanged, individual syncs may queue briefly.
  • Any hit the default maxLine cap WARN identifies a sheet that needs pagination or an explicit limit.

@Danswar

Danswar commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

4 review passes to zero findings. Fixed along the way: worker slots could wedge permanently on a never-settling query (item timeout added); timed-out queued exports no longer execute after their caller already got an error (load-shedding in the shared queue util, with new spec coverage); the default maxLine cap moved from the DTO into the service so truncation is detected and WARN-logged per bank_tx sub-query instead of on the concatenated total; .take() on the three joined bank_tx raw sub-queries emitted no SQL LIMIT at all (verified via getSql()) and was replaced with .limit(), making the per-sub-query cap real; /gs/db/custom now reports failures as 400 like /gs/db; plus test-hygiene cleanups (typed internals bridge, queue teardown in all suites).

@Danswar
Danswar marked this pull request as ready for review July 23, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant