perf(gs): queue sheet exports, cap unbounded queries and monitor event-loop delay#4346
perf(gs): queue sheet exports, cap unbounded queries and monitor event-loop delay#4346Danswar wants to merge 4 commits into
Conversation
…ault cap, consistent 400 on custom exports
… false-positive warns
…e emits no LIMIT with joins)
|
4 review passes to zero findings. Fixed along the way: worker slots could wedge permanently on a never-settling query (item timeout added); timed-out queued exports no longer execute after their caller already got an error (load-shedding in the shared queue util, with new spec coverage); the default maxLine cap moved from the DTO into the service so truncation is detected and WARN-logged per bank_tx sub-query instead of on the concatenated total; |
Problem
Part of #4227. Investigation of the reported internal-tooling slowness (07-21/22) showed that the dominant share of slow
/gs/dbrequests is not DB work: queries returning 0 rows from tiny tables (e.g.country, 250 rows) took 5–7.5s because sheet-export syncs fire in aligned bursts and their result post-processing runs synchronously on the event loop. Every in-flight request — including interactive support/compliance calls — absorbs those stalls. Additionally, ~22% of slow exports set nomaxLineat all (the DTO leaves it optional), so unbounded full-table pulls are possible by omission.Changes
GsService):POST /gs/dbandPOST /gs/db/customnow run through aQueueHandlerwithmaxWorkParallel = 2, a 240s queue timeout (below the Apps-Script 6-min execution cap) and a 240s item timeout (a query that never settles frees its worker slot instead of wedging the queue). Exports are latency-tolerant batch consumers; interactive requests no longer compete with a whole burst at once. Both endpoints report failures (incl. queue timeouts) consistently as HTTP 400 —/gs/db/custompreviously surfaced errors as opaque 500s.QueueHandlerload-shedding (shared util): items whose queue timeout fired while still waiting are discarded instead of executed — previously their action ran anyway and the result was silently dropped, doubling load exactly when the queue is backed up. Covered by a newqueue-handler.spec.ts.maxLinedefault cap (GsService): requests withoutmaxLine(absent ornull) now run with a cap of 10000 rows instead of unlimited (on the custom bank_tx export the cap applies per sub-query, and the truncation signal is derived per sub-query). Explicit values are untouched (the escape hatch for deliberate large exports, which also keeps the existing >100k alert mail reachable), and the largest observed no-limit export returns ~1.4k rows, so no existing consumer changes behavior. When a default-capped export actually hits the cap, a WARN log names the identifier and table — truncation is visible, not silent.MonitorEventLoopService): logsmean / p95 / maxevent-loop delay every 10s (same cadence/level as the existing connection-pool monitor, gated by a newProcess.MONITOR_EVENT_LOOP). Makes the next slowness report attributable in one log query instead of an elimination hunt.Tests
gs.service.spec.ts: queue caps concurrency at 2 and preserves per-call results; custom exports route through the same queue; default cap applies for absent andnullmaxLine, explicit values pass through; queue loops stopped in teardown.queue-handler.spec.ts(new): normal execution, discarded-after-timeout items never run, item timeout frees a wedged slot.Verification after deploy
EventLoop delaylines appear in the API log; during sheet-sync bursts p95 shows the stall magnitude.msvalues) during burst minutes should drop;/gs/dbthroughput is unchanged, individual syncs may queue briefly.hit the default maxLine capWARN identifies a sheet that needs pagination or an explicit limit.