perf(support): cut dashboard overview and search query time

[joshuakrueger-dfx]

Problem

Two support-dashboard endpoints were slow, and the cost was in how they queried — not in the amount of data. Measured against a seeded copy at production cardinality (8.6k support_issue, 94.6k support_message, 8.6k accounts), median of 7 runs through the real service methods:

	before
Overview (all open tickets)	204 ms
Ticket list with a search term	106 ms

Overview — 76% of it was getMessageStats, which asked "what is the newest message?" with two correlated subqueries per issue. At the ~1360 open tickets the overview loads, that is ~2720 subquery executions over the same rows the outer query already scanned.

Search — 62 of the 111 ms were the getManyAndCount total alone. Each search term expands to ... OR EXISTS (SELECT 1 FROM support_message m WHERE m."issueId" = issue.id AND m.message LIKE '%term%'), and a leading-wildcard LIKE cannot use a b-tree index, so every message row was read to produce the count — again on every "load more".

Change

1. getMessageStats in one pass — the newest message is picked inside the same GROUP BY via (array_agg(... ORDER BY m.id DESC))[1] instead of two correlated subqueries. Same ordering semantics (by id, not created).
2. Trigram index on support_message.message (new migration) so the search predicate and its total stop reading every message row.
3. messageThreadQuery helper for the three thread reads (closeIssue, getIssue, getIssueMessages): adds loadEagerRelations: false and an explicit ORDER BY id ASC.
4. Missing order in the auto-responder — SupportIssueJobService.getAutoResponseIssues filters on messages.at(-1).author, i.e. it indexes into the loaded array positionally, but loaded the thread unordered. Its sibling autoOnHold has always ordered. Without it the bot decides "the customer wrote last" off an arbitrary row, so it can mail tickets support already answered and skip ones actually waiting.

Result

	before	after
Overview (all open tickets)	204 ms	97 ms
Ticket list with a search term	106 ms	20 ms
Statistics, 365 days	68 ms	43 ms
Activity poll (every 30 s)	36 ms	19 ms

getMessageStats in isolation: 372 ms → 23 ms for the ~1360 open issues (47 ms → 6 ms at 200; at 20 it barely matters). Search getManyAndCount: 111 ms → 18 ms, of which the total dropped 62 ms → 7 ms.

Equivalence of the rewritten aggregate was verified against the real database rather than argued — identical count, lastDate and lastAuthor at 20 / 200 / ~1360 issues, including the case where the newest id carries the older created.

Only support_message.message is indexed. Trigram indexes on issue.name, user_data.firstname and surname were measured too and moved the combined query by ~3 ms, which does not justify three more GIN indexes.

Migration

1784900000000-AddSupportMessageTrigramIndex

Migrations run at boot inside a single transaction, so any raw failure here rolls the whole batch back and the API does not start — an outage caused by a pure performance index. A privilege check alone does not cover that: the extension may be absent from the server, blocked by a managed provider's allow-list, or installed into a schema outside the search_path, in which case pg_extension reports it as present while gin_trgm_ops still does not resolve.

Both DDL statements therefore run inside PL/pgSQL EXCEPTION blocks. Catching in JS would not help — a failed statement poisons the Postgres transaction — but a PL/pgSQL exception block opens an implicit subtransaction, so the failure is contained. query_canceled is listed explicitly because Postgres excludes it from WHEN OTHERS; a server-side statement_timeout shorter than the index build would otherwise abort the batch anyway. A lock_timeout bounds the wait if a writer already holds a conflicting lock, and is reset afterwards so it does not leak onto later migrations in the same batch.

Every state was exercised against a real Postgres:

state	outcome
extension present	index created
absent, role may create it	extension + index created
absent, role may not	skipped, NOTICE, boot unaffected
present but opclass unresolvable	skipped, NOTICE, boot unaffected
lock or statement timeout	skipped, NOTICE, boot unaffected
second run	idempotent

If that NOTICE appears, nothing retries — TypeORM records the migration as applied either way. Installing the extension afterwards needs a follow-up migration that creates the index alone. Worth checking before merge: SELECT * FROM pg_extension WHERE extname = 'pg_trgm';

Cost of catching query_canceled: an operator's pg_cancel_backend() aimed at a slow build is swallowed too; cancelling this statement specifically needs pg_terminate_backend.

Plain CREATE INDEX, not CONCURRENTLY — that is not allowed inside a transaction, and overriding the transaction mode throws ForbiddenTransactionModeOverrideError while the batch runs under all. The lock is a ShareLock: SELECT keeps working, INSERT blocks. Because the batch is one transaction it is held until the whole batch commits, not just for the ~4-6 s build (GIN ~16 MB next to a ~13 MB table). New support and customer messages cannot be written for that window — a reason to ship this migration on its own if a release carries several slow ones.

Also fixed on the same paths

Message threads had no ORDER BY at all, and SupportIssueJobService.getAutoResponseIssues picked the last message with messages.at(-1) over that unordered array — so the auto-responder could mail tickets support had already answered and skip ones that were waiting. Its sibling autoOnHold had always ordered. Both now order by created with an id tie-break.

The same filter also threw on an issue whose first message was never written (createIssueInternal commits the issue and its first message in separate transactions). The cron lock caught and logged it, but aborted the rest of the run — every minute, until someone read the log.

Tests

• New support-message-trigram-index.migration.spec.ts — runs the real migration against a throwaway Postgres (MIGRATION_TEST_PG, skipped without a DB like the sibling suites): index shape, down after a skipped up, idempotency, and — the point of the whole construction — that a failing CREATE INDEX leaves the migration batch usable. Two cases need database-global DDL (dropping the extension, creating a role); those are opt-in via MIGRATION_TEST_PG_EXCLUSIVE so they cannot perturb the ~10 real-Postgres suites sharing one database in CI.
• New message-stats-aggregate.spec.ts — executes the aggregate expressions the service itself exports, not a copy, against a real Postgres: newest-by-id semantics, NULL handling, and row-for-row equality with the correlated-subquery form it replaced.
• New support-issue-job.service.spec.ts — pins the ordering both auto-response call sites depend on, and the empty-thread guard.
• Extended service spec for the scoped/ordered thread reads.

Every new test was checked in the other direction: each one was run against the code with its fix removed and confirmed to fail. Two earlier versions of these tests passed for the wrong reason and were rewritten.

Scope

DFX support board only. The RealUnit tenant screen carries the same class of defect and is deliberately not part of this PR.

Release Checklist

Pre-Release

• Check migrations
  • No database related infos (sqldb-xxx)
  • Impact on GS (new/removed columns): none — index only, no schema change
• Check for linter errors (in PR)
• Test basic user operations (on DFX services)
  • Login/logout
  • Buy/sell payment request
  • KYC page

Post-Release

• Test basic user operations
• Monitor application insights log