feat(github-app): allow several github apps to be used

[thomasnemer]

Summary

• Add support for configuring additional GitHub Apps to distribute API calls across multiple rate limit buckets, making the module scalable for large enterprise deployments
• New additional_github_apps variable (optional, no breaking changes) accepts a list of extra apps with id, key_base64, and optionally installation_id (all support direct values or SSM references)
• Lambda functions randomly select an app per authentication flow and thread the selection through the entire JWT → installation token → API call chain for consistency
• Installation ID resolution is optimized: primary app reuses the webhook payload, additional apps use a pre-configured installation_id when provided, falling back to an API lookup only when needed

Context

At enterprise scale with many runner configurations, all Lambda functions (scale-up, scale-down, pool, job-retry) share a single GitHub App's rate limit bucket (15,000 req/hour on GitHub Enterprise Cloud). This ceiling cannot be raised per-app — the only way to scale is to spread load across multiple apps. This change makes the effective limit N × 15,000 req/hour, without introducing breaking changes.

Closes #5037

[edersonbrilhante]

@thomasnemer can you fix the conflicts?

[thomasnemer]

For sure. I'll rebase early next week.

[thomasnemer]

@edersonbrilhante done :)

[npalm]

@thomasnemer can you check the failing pipepline?

[thomasnemer]

@npalm done :) (+ rebased and resolved a conflict on scale-up.ts)

[thomasnemer]

@npalm : How confident are we with merging this PR for it to be part of the next release? Not pushing forward at all, just planning for internal maintenance of our non-production environment :)

[thomasnemer]

@edersonbrilhante 👋 Is there anything preventing this PR to go forward that I can help with?

[gfliker]

bumping this.
Thanks

[edersonbrilhante]

@thomasnemer the project got more maintainers to speed the reviews. Can you fix the conflicts?

[thomasnemer]

@edersonbrilhante sorry I missed the ping last week 🙇. The PR has been rebased and conflicts solved.

[edersonbrilhante]

@thomasnemer Sorry to ask again, can you fix the conflicts? Next review I will prioritise your PR :)

[thomasnemer]

@thomasnemer Sorry to ask again, can you fix the conflicts? Next review I will prioritise your PR :)

It's rebased and conflict free now @edersonbrilhante :)

[edersonbrilhante]

@thomasnemer I will this in my environment during this week. Are you using this feature in your environment?

[thomasnemer]

@thomasnemer I will this in my environment during this week. Are you using this feature in your environment?

@edersonbrilhante Yes, in our staging and production environments, with 3 gh apps (main one + 2 additional) to balance the load.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

lambdas/functions/control-plane/src/github/rate-limit.ts:33

• TypeScript type mismatch: metricGitHubAppRateLimit passes an optional appIndex (number | undefined) into getAppId(), which currently expects a number. This fails compilation under strictNullChecks and also risks passing undefined through to SSM lookup logic.

    const updateMetric = yn(process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT);
    if (updateMetric) {
      const appId = await getAppId(appIndex);
      const metric = createSingleMetric('GitHubAppRateLimitRemaining', MetricUnit.Count, remaining, {
        AppId: appId,

modules/runners/README.md:165

• The input documentation says "round-robin" distribution, but the implementation (and the rest of this paragraph) describes random selection per auth flow. This is contradictory and can confuse users about expected behavior.

| <a name="input_github_app_parameters"></a> [github\_app\_parameters](#input\_github\_app\_parameters) | Parameter Store for GitHub App Parameters.<br/><br/>Supports multiple GitHub Apps for round-robin API rate limit distribution.<br/>Each list element corresponds to one GitHub App and is a map containing<br/>`name` and `arn` keys referencing SSM parameters. The first element is the<br/>primary app (the one whose webhook secret is used for incoming webhook<br/>validation). All apps must be installed on the same repositories/organizations.<br/><br/>The control-plane lambdas (scale-up, scale-down, pool, job-retry) randomly<br/>select an app from the list for each GitHub API call, distributing rate<br/>limit consumption across all configured apps. | <pre>object({<br/>    key_base64      = list(map(string))<br/>    id              = list(map(string))<br/>    installation_id = list(object({ name = string, arn = string }))<br/>  })</pre> | n/a | yes |

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

modules/runners/README.md:165

• This row says the module supports "round-robin" distribution, but the implementation in lambdas/functions/control-plane/src/github/auth.ts selects an app via Math.floor(Math.random() * credentials.length) (random selection), not deterministic round-robin. Update wording to avoid implying deterministic ordering.

| <a name="input_github_app_parameters"></a> [github\_app\_parameters](#input\_github\_app\_parameters) | Parameter Store for GitHub App Parameters.<br/><br/>Supports multiple GitHub Apps for round-robin API rate limit distribution.<br/>Each list element corresponds to one GitHub App and is a map containing<br/>`name` and `arn` keys referencing SSM parameters. The first element is the<br/>primary app (the one whose webhook secret is used for incoming webhook<br/>validation). All apps must be installed on the same repositories/organizations.<br/><br/>The control-plane lambdas (scale-up, scale-down, pool, job-retry) randomly<br/>select an app from the list for each GitHub API call, distributing rate<br/>limit consumption across all configured apps. | <pre>object({<br/>    key_base64      = list(map(string))<br/>    id              = list(map(string))<br/>    installation_id = list(object({ name = string, arn = string }))<br/>  })</pre> | n/a | yes |

modules/multi-runner/README.md:9

• The section title/wording says "round-robin", but the lambdas select a GitHub App randomly (see createGithubAppAuth in lambdas/functions/control-plane/src/github/auth.ts). Consider renaming the section to avoid suggesting deterministic round-robin.

### GitHub App round-robin

To distribute GitHub API rate limit usage, this module supports configuring multiple GitHub Apps via the `additional_github_apps` variable. The control-plane lambdas (scale-up, scale-down, pool, job-retry) randomly select an app for each API call, spreading the load across all configured apps.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

lambdas/functions/control-plane/src/scale-runners/scale-down.ts:61

• The cached Octokit client is not tagged with which GitHub App index was used to authenticate it. Downstream calls (e.g., rate-limit metrics) therefore can’t attribute requests to the correct app, which makes multi-app rate-limit metrics misleading.

  const ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl, appIdx);
  const octokit = await createOctokitClient(ghAuth.token, ghesApiUrl);
  githubCache.clients.set(key, octokit);

lambdas/functions/control-plane/src/scale-runners/scale-down.ts:85

• In multi-app mode, this metric call always defaults to appIndex=0, even when the Octokit client was authenticated with a different app. This will misattribute rate-limit telemetry and can hide which app is actually being throttled.

    // appIndex is not threaded here: the octokit client is cached per-owner and does not
    // retain which app authenticated it, so the rate-limit metric labels the primary app (index 0).
    metricGitHubAppRateLimit(state.headers);

modules/runners/README.md:165

• This input description says “round-robin API rate limit distribution”, but the implementation and the rest of the text describe random selection. To avoid confusing users, please use consistent terminology (e.g., “random” / “load-balanced”).

| <a name="input_github_app_parameters"></a> [github\_app\_parameters](#input\_github\_app\_parameters) | Parameter Store for GitHub App Parameters.<br/><br/>Supports multiple GitHub Apps for round-robin API rate limit distribution.<br/>Each list element corresponds to one GitHub App and is a map containing<br/>`name` and `arn` keys referencing SSM parameters. The first element is the<br/>primary app (the one whose webhook secret is used for incoming webhook<br/>validation). All apps must be installed on the same repositories/organizations.<br/><br/>The control-plane lambdas (scale-up, scale-down, pool, job-retry) randomly<br/>select an app from the list for each GitHub API call, distributing rate<br/>limit consumption across all configured apps. | <pre>object({<br/>    key_base64      = list(map(string))<br/>    id              = list(map(string))<br/>    installation_id = list(object({ name = string, arn = string }))<br/>  })</pre> | n/a | yes |

modules/multi-runner/README.md:7

• The section title says “round-robin”, but the behavior described here (and implemented in the lambdas) is random selection. Renaming this heading avoids implying a deterministic rotation strategy.

### GitHub App round-robin

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

lambdas/functions/control-plane/src/github/auth.ts:212

• createAuth can throw a hard-to-diagnose runtime error if no app credentials are configured (empty PARAMETER_* env vars) or if an out-of-range appIndex is ever passed. Today selected can become undefined and the next line will crash when accessing selected.appId. Add explicit validation so failures are deterministic and actionable.

  const credentials = await getAppCredentials();
  const selected =
    appIndex !== undefined ? credentials[appIndex] : credentials[Math.floor(Math.random() * credentials.length)];

  logger.debug(`Selected GitHub App ${selected.appId} for authentication`);

lambdas/functions/control-plane/src/scale-runners/scale-down.ts:84

• Rate-limit metrics are now potentially misattributed in multi-app mode: the Octokit client may be authenticated with a non-primary app, but metricGitHubAppRateLimit is called without the corresponding appIndex so it will default to index 0. This makes the new per-app metric labeling unreliable.

    // appIndex is not threaded here: the octokit client is cached per-owner and does not
    // retain which app authenticated it, so the rate-limit metric labels the primary app (index 0).
    metricGitHubAppRateLimit(state.headers);

modules/runners/README.md:165

• This description says "round-robin" but the implementation selects the app randomly (see Math.random() in lambdas/functions/control-plane/src/github/auth.ts). The docs should match the actual selection strategy.

| <a name="input_github_app_parameters"></a> [github\_app\_parameters](#input\_github\_app\_parameters) | Parameter Store for GitHub App Parameters.<br/><br/>Supports multiple GitHub Apps for round-robin API rate limit distribution.<br/>Each list element corresponds to one GitHub App and is a map containing<br/>`name` and `arn` keys referencing SSM parameters. The first element is the<br/>primary app (the one whose webhook secret is used for incoming webhook<br/>validation). All apps must be installed on the same repositories/organizations.<br/><br/>The control-plane lambdas (scale-up, scale-down, pool, job-retry) randomly<br/>select an app from the list for each GitHub API call, distributing rate<br/>limit consumption across all configured apps. | <pre>object({<br/>    key_base64      = list(map(string))<br/>    id              = list(map(string))<br/>    installation_id = list(object({ name = string, arn = string }))<br/>  })</pre> | n/a | yes |

modules/multi-runner/README.md:7

• The docs call this "round-robin" but the implementation selects apps randomly (see Math.random() in lambdas/functions/control-plane/src/github/auth.ts). Renaming the section avoids implying deterministic rotation.

### GitHub App round-robin

modules/multi-runner/README.md:13

• This bullet says "round-robin" but selection is random (see Math.random() in lambdas/functions/control-plane/src/github/auth.ts). Consider adjusting wording to match behavior.

- Its **app ID and private key** are included in the round-robin pool alongside the additional apps.

modules/multi-runner/README.md:17

• This sentence says "round-robin" but the control-plane uses random selection (see Math.random() in lambdas/functions/control-plane/src/github/auth.ts). Align terminology to prevent operator confusion.

The **webhook lambda** does not participate in round-robin: it only validates incoming webhook signatures using the primary app's webhook secret and never calls the GitHub API.

[thomasnemer]

@edersonbrilhante Are we good to go?

[edersonbrilhante]

@thomasnemer Can you fix? Heads up, I am working to refactor the module to introduce plugin layout. Ref: #5234

[Brend-Smits]

I've actually been testing this change on my fork and it's working very well so far! Would love to get this merged as soon as possible as it's something lots of people would benefit from.
It's a bit difficult though as we can't make edits on yours branch.
Have a look at solving the merge conflicts, then let's merge this.

[guicaulada]

Great to see this approved, we want this capability upstream too. We've been running a nearly identical implementation in production (vendored on v7.10.0: same per-flow app selection, stored installation ids, and SSM/IAM layout), so these notes come from operating this design at scale.

One thing we think should be fixed before this ships in a release:

The PR removes the 404 stale-installation fallback (createGithubInstallationClient in scale-up.ts, and the try/catch in octokit.ts's getOctokit). That fallback protects existing single-app deployments today: when a webhook payload carries a stale installationId (app uninstalled/reinstalled while messages were in flight), the current code catches the 404, re-resolves the installation via the API, and retries. With this PR the primary-app path trusts payload.installationId and the 404 propagates, so every scale-up for that repo fails until stale payloads drain from the queue. It's rare, but silent and hard to diagnose.

The restore is small and composes with app selection, the catch block can re-resolve using the already-selected appIndex. Since the conflict rebase touches these files anyway, that would be a good time to fix it. If you'd rather keep this PR as-is, we're happy to open the restoring PR ourselves right after merge, we'd just ask it get's in before the next release is cut.

Things we'd suggest as follow-ups (none blocking, we can contribute the first two from our production fork):

1. Manifest parameter instead of colon-joined env vars. PARAMETER_GITHUB_APP_*_NAME grows linearly with app count and Lambda's total environment limit is 4 KB — with typical parameter path lengths that caps out around 15–20 apps, which matters for the large-enterprise use case this PR targets. A small SSM manifest parameter listing the per-app parameter names keeps the env constant regardless of app count. Internal contract only (TF and lambdas deploy together), so changing it later breaks nothing.
2. Rate-limit-aware app selection. Uniform random keeps sending ~1/N of traffic to an app that's already throttled. We track x-ratelimit-remaining per app from response headers, put apps in a short cooldown on secondary rate limits, and pick the app with the most budget left (random start offset so cold starts don't converge). Selection is internal, so this slots in cleanly later.
3. Validate installation-id list alignment in loadAppCredentials: id/key counts are checked, but the positionally-aligned installation-id list isn't; drift silently shifts installation ids across apps.
4. Reuse loaded credentials in rate-limit.ts:getAppId re-reads SSM per app and throws on out-of-range indexes inside a swallowed try/catch, silently stopping the metric.
5. Doc callout that additional apps must be installed on the same orgs/repos: a misconfigured app today surfaces only as intermittent installation 404s, which random selection makes especially confusing.

We'll open PRs for the follow-ups we plan to pick up, so nothing here needs to hold up the merge beyond the rebase.