Skip to content

feat(webhook): use priority field as tiebreaker in matcher dispatch - #5133

Closed
vegardx wants to merge 1 commit into
github-aws-runners:mainfrom
vegardx:feat/priority-dispatch-matcher-config
Closed

feat(webhook): use priority field as tiebreaker in matcher dispatch#5133
vegardx wants to merge 1 commit into
github-aws-runners:mainfrom
vegardx:feat/priority-dispatch-matcher-config

Conversation

@vegardx

@vegardx vegardx commented May 26, 2026

Copy link
Copy Markdown
Contributor

Problem

When multiple runner pools match the same job labels via exactMatch: true (subset matching), dispatch order is non-deterministic. A workflow requesting [self-hosted, linux, x64] can land on any pool whose labels contain that subset — including an expensive 16 vCPU on-demand pool when a cheap 2 vCPU spot pool would suffice.

See #5128 for the full discussion and concrete examples.

Fix

The Terraform variable runner_matcher_config already supports an optional priority field (default 999) and pre-sorts the config before writing to SSM. However the Lambda re-sorts only by exactMatch, discarding the priority ordering. This PR adds priority as a secondary sort key:

matcherConfig.sort((a, b) => {
  if (a.matcherConfig.exactMatch !== b.matcherConfig.exactMatch) {
    return a.matcherConfig.exactMatch ? -1 : 1;
  }
  return (a.matcherConfig.priority ?? 999) - (b.matcherConfig.priority ?? 999);
});

The assumption is that when a workflow omits specific labels (e.g. uses just [self-hosted, linux, x64] instead of [self-hosted, linux, x64, large, ondemand]), the omission is intentional — the requester does not care which specific pool handles the job. The priority field lets the operator decide which pool should be preferred for these ambiguous requests.

Changes

  • lambdas/functions/webhook/src/sqs/index.ts — add optional priority field to MatcherConfig
  • lambdas/functions/webhook/src/runners/dispatch.ts — sort by priority as tiebreaker
  • lambdas/functions/webhook/src/runners/dispatch.test.ts — test: multiple pools match common subset, lowest priority wins

Backwards compatible

  • priority is optional, defaults to 999
  • Existing configs without priority behave identically (all sort equal, first match wins as before)
  • The Terraform variable already defines priority = optional(number, 999) — no Terraform changes needed

Refs: #5128

When multiple runner pools match the same job labels (subset match via
exactMatch: true), sort by ascending priority so the lowest-priority
pool (cheapest) is selected first.

The Terraform variable already supports priority (optional, default 999)
and pre-sorts the config before writing to SSM. However the Lambda
re-sorted only by exactMatch, losing the priority ordering. This commit
adds priority as a secondary sort key in the Lambda.

Refs: github-aws-runners#5128
@vegardx

vegardx commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Closing this one — I was wrong about the premise, and main has moved past it.

priority already works end-to-end: modules/webhook/webhook.tf builds the matcher keys as format("%03d-%s", priority, k) and sorts them before serialising to SSM, so the array reaching the lambda is already in priority order. Re-sorting in the lambda just redoes Terraform's work. And Array.sort has been stable by spec since ES2019, so the tiebreaker this PR adds isn't fixing real nondeterminism.

Worse, the comparator here predates bidirectionalLabelMatch (#5031) — merging it as-is would regress bidirectional matching back to exactMatch only.

What was worth salvaging is in #5214: declaring priority on the MatcherConfig type, and not sorting the shared config in place — ConfigLoader holds it on a static singleton, so the in-place sort lets one event reorder what the next one sees on a warm container. Idempotent today, but it makes dispatch order depend on invocation history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant