feat(multi-runner): add experimental runner provider config v2 - #5251
Open
edersonbrilhante wants to merge 7 commits into
Open
feat(multi-runner): add experimental runner provider config v2#5251edersonbrilhante wants to merge 7 commits into
edersonbrilhante wants to merge 7 commits into
Conversation
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
edersonbrilhante
force-pushed
the
experimental-multi-runner-config-v2-20260805
branch
from
August 5, 2026 20:58
34b9f68 to
eafd4b5
Compare
Brend-Smits
reviewed
Aug 6, 2026
Brend-Smits
left a comment
Brend-Smits
left a comment
Contributor
There was a problem hiding this comment.
I like the general direction where this is going. Wondering if it makes sense to introduce some kind of schema for people to consume as well now that we are making these changes.
This will be a much needed change seeing as we are moving to a new architecture that supports multiple compute variants. Please do keep in mind though that the people that are reading this documentation (like in your variables.tf), might now have all the context that you have. So try to keep the easy to consume for those folks as well (explain jargon, etc).
Do I understand it correctly that this change will be opt-in for this version while we test it further and in the future we will make a breaking change version update to change the default to this new config?
Great work!
edersonbrilhante
changed the title
This was referenced Aug 6, 2026
…dules (#5257) ## Description - Keep `modules/runners` and stable `multi_runner_config` dispatch unchanged. Stable configurations retain their historical `module.runners["configuration"]` addresses and flat `runners_map` fields. - Add explicit opt-in through `experimental.multi_runner_config_v2`. Stable and experimental configurations can coexist when their keys do not overlap; duplicate keys are rejected. - Normalize stable v1 once for shared queues, webhook matching, and runner-binary discovery while routing only v2 configurations through `modules/runner-stack`. - Make `runner-stack` the provider-neutral control plane for scale-up, scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and runner-role ownership. - Keep EC2-specific launch templates, instance profiles, security groups, AMI/bootstrap resources, runner log groups, IAM fragments, and Lambda environment fragments under `modules/compute-providers/ec2`. - Define provider-owned runner-role requirements in EC2 and attach them to the common runner role in `runner-stack`, allowing future compute providers to supply different policies without duplicating the role lifecycle. - Replace flat runner-stack inputs with ownership-based nested objects. Logging configuration is grouped under `observability.logs`, including `level`, retention, encryption, class, and tags. - Pass the canonical `compute_provider.ec2` object and nested `runner`, `github`, `ssm`, and `observability` objects directly into the EC2 resource and runner-role policy modules instead of expanding them back into prefixed scalar inputs. - Layer module, shared-resource, component, subcomponent, and EC2 runtime tags with documented precedence; provider-required EC2 bootstrap tags retain final precedence. - Group experimental v2 outputs by ownership: `runner.role`, `scale_up.{lambda,log_group,role}`, `scale_down.{lambda,log_group,role}`, nullable `pool.{lambda,log_group,role}`, and provider-specific resources under `provider.<type>`. - Use caller-known optional wrappers for external AMI parameters and KMS keys. The wrapper determines Terraform graph shape while its `arn` leaf may remain unknown until apply. - Generate runner-stack, pool, job-retry, and EC2 IAM policies with `aws_iam_policy_document` and retain provider-policy merge behavior. - Document the experimental boundary, ownership model, plan-time wrapper pattern, phased migration, and nested output contract under the internal module documentation path. This draft is stacked on #5251 because the provider boundary consumes the experimental v2 normalization introduced there. Lambda/TypeScript terminology changes are tracked separately in #5258. ## Test Plan - `pre-commit run --all-files` — Terraform fmt, TFLint, validation, and merge-conflict checks passed. - `terraform test` in `modules/runner-stack` — 11 passed. - `terraform test` in `modules/multi-runner` — 7 passed. - `terraform test` in `modules/compute-providers/ec2` — 4 passed. - `terraform test` in `modules/compute-providers/ec2/runner-role` — 3 passed. - `terraform test` in `modules/runner-stack/pool` — 1 passed. - `terraform test` in `modules/runner-stack/job-retry` — 1 passed. - `terraform validate` in `modules/lambda` — passed. - Verified `modules/runners` has no diff from `origin/main`, stable v1 still dispatches only to `module.runners`, and only the experimental map dispatches to `module.runner_stacks`. - Verified computed external role, profile, AMI-parameter, managed-policy, and KMS ARN inputs plan successfully through the real wrapper fixture. No live AWS apply was performed. Terraform tests use mocked providers, and state migration is intentionally deferred to the later migration phase. ## Related Issues Closes #5252 Depends on #5251 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Contributor
Author
|
@Brend-Smits I changed my first implementation. I will not touch runners. Instead I created a whole cloned runners, refactoring it to add support for a decoupled ec2 module. |
edersonbrilhante
marked this pull request as ready for review
August 7, 2026 20:19
edersonbrilhante
force-pushed
the
experimental-multi-runner-config-v2-20260805
branch
from
August 7, 2026 21:42
4a9c778 to
0849752
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an experimental
multi_runner_config_v2input for the multi-runner module while keepingmulti_runner_configas the stable EC2 lane contract. The module now normalizes v1 into the v2 internal lane model unless v2 is provided, and uses the normalized shape for queues, webhook matcher config, and EC2 runner module inputs.MicroVM and CodeBuild are left as commented future-provider references only; Terraform support remains EC2-only in this PR. This is stacked on #5250.
Also adds a targeted TFLint suppression for the existing module-level
iam_overridesinput, which is retained as public module interface but is not currently consumed by module resources.Test Plan
terraform fmt -check modules/multi-runner modules/webhookgit diff --checkterraform -chdir=modules/multi-runner validateRelated Issues
N/A