Skip to content

fix(scale-up): return partial runner creation results - #5220

Merged
edersonbrilhante merged 7 commits into
mainfrom
refactor-partial-batch-errors
Jul 30, 2026
Merged

fix(scale-up): return partial runner creation results#5220
edersonbrilhante merged 7 commits into
mainfrom
refactor-partial-batch-errors

Conversation

@edersonbrilhante

@edersonbrilhante edersonbrilhante commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Description

Refactor runner creation to return structured partial-success results instead of throwing ScaleError.

  • Return created instance IDs with retryableErrorCount and nonRetryableErrorCount.
  • Preserve successfully created EC2 instances when Fleet or dedicated-host requests only partially succeed.
  • Classify configured scale errors, transient AWS errors, network failures, throttling, and server failures as retryable.
  • Treat unclassified AWS errors as non-retryable.
  • Propagate the result contract through EC2, pool, and scale-up providers.
  • Convert unexpected provider exceptions into retryable result counts.
  • Treat GitHub runner registration failures as retryable and terminate the affected EC2 instances.
  • Retry only the corresponding number of SQS messages.
  • Remove the obsolete ScaleError implementation and handler logic.
  • Skip unsupported event types without retrying them.

Test Plan

TBD

Related Issues

None.

@edersonbrilhante
edersonbrilhante requested a review from a team as a code owner July 22, 2026 21:40
@edersonbrilhante
edersonbrilhante marked this pull request as draft July 22, 2026 21:40
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Comment thread lambdas/functions/control-plane/src/lambda.ts Outdated
@wadherv

wadherv commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

@npalm This fix is quite critical for us. The recent GitHub outages have significantly impacted our workflows, resulting in much higher queue times. Would it be possible to get a release out with these changes as soon as possible?

Thanks!

@edersonbrilhante

Copy link
Copy Markdown
Contributor Author

This PR depends on the change I did on #5203.

Base automatically changed from refactor-runner-provider-split to main July 30, 2026 17:50
@edersonbrilhante
edersonbrilhante requested a review from a team as a code owner July 30, 2026 17:50
@edersonbrilhante
edersonbrilhante force-pushed the refactor-partial-batch-errors branch 2 times, most recently from 69367f3 to 1e016e3 Compare July 30, 2026 19:03
@edersonbrilhante
edersonbrilhante force-pushed the refactor-partial-batch-errors branch from 1e016e3 to fcee6cb Compare July 30, 2026 19:05
@edersonbrilhante
edersonbrilhante merged commit 9045340 into main Jul 30, 2026
9 checks passed
@edersonbrilhante
edersonbrilhante deleted the refactor-partial-batch-errors branch July 30, 2026 19:47
edersonbrilhante pushed a commit that referenced this pull request Jul 31, 2026
🤖 I have created a release *beep* *boop*
---


##
[7.10.1](v7.10.0...v7.10.1)
(2026-07-31)


### Bug Fixes

* **control-plane:** discard malformed SQS messages
([#5219](#5219))
([8ca5816](8ca5816))
* enable Octokit throttle retry and SSM adaptive retry under burst
([#5136](#5136))
([cbdd916](cbdd916)),
closes
[#5135](#5135)
* **lambda:** bump axios from 1.17.0 to 1.18.0 in /lambdas
([#5211](#5211))
([67bfdf8](67bfdf8))
* **lambda:** bump js-yaml from 3.14.2 to 3.15.0 in /lambdas
([#5212](#5212))
([facea4e](facea4e))
* **lambda:** bump postcss from 8.5.15 to 8.5.22 in /lambdas
([#5231](#5231))
([8ee045d](8ee045d))
* **lambda:** fail-open isJobQueued — assume queued on API errors
([#5130](#5130))
([8fa06ee](8fa06ee))
* **scale-up:** return partial runner creation results
([#5220](#5220))
([9045340](9045340))
* stop a malformed SQS body from poisoning the whole batch
([#5215](#5215))
([69e06b1](69e06b1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants