-
Notifications
You must be signed in to change notification settings - Fork 0
feat(buildrunner): add GitHub Actions backend #208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
albertywu
wants to merge
2
commits into
main
Choose a base branch
from
wua/github-actions-buildrunner
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
30 changes: 30 additions & 0 deletions
30
submitqueue/extension/buildrunner/githubactions/BUILD.bazel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| load("@rules_go//go:def.bzl", "go_library", "go_test") | ||
|
|
||
| go_library( | ||
| name = "githubactions", | ||
| srcs = [ | ||
| "client.go", | ||
| "githubactions.go", | ||
| ], | ||
| importpath = "github.com/uber/submitqueue/submitqueue/extension/buildrunner/githubactions", | ||
| visibility = ["//visibility:public"], | ||
| deps = [ | ||
| "//submitqueue/entity", | ||
| "//submitqueue/extension/buildrunner", | ||
| "@org_uber_go_zap//:zap", | ||
| ], | ||
| ) | ||
|
|
||
| go_test( | ||
| name = "githubactions_test", | ||
| srcs = ["githubactions_test.go"], | ||
| embed = [":githubactions"], | ||
| deps = [ | ||
| "//core/httpclient", | ||
| "//submitqueue/entity", | ||
| "//submitqueue/extension/buildrunner", | ||
| "@com_github_stretchr_testify//assert", | ||
| "@com_github_stretchr_testify//require", | ||
| "@org_uber_go_zap//:zap", | ||
| ], | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # GitHub Actions BuildRunner | ||
|
|
||
| `githubactions` implements `buildrunner.BuildRunner` with GitHub Actions | ||
| `workflow_dispatch`. It is intended to prove the SubmitQueue BuildRunner | ||
| architecture against a common CI system without adding local state. | ||
|
|
||
| ## How it works | ||
|
|
||
| 1. `Trigger` dispatches the configured workflow on a trusted ref, usually | ||
| `main`, and returns GitHub's workflow run ID as the SubmitQueue build ID. | ||
| 2. SubmitQueue passes these workflow inputs: | ||
| - `sq_base_uris` | ||
| - `sq_head_uris` | ||
| - `sq_queue` | ||
| - `sq_metadata` | ||
| 3. `Status` calls GitHub's get-workflow-run endpoint with that run ID. | ||
| 4. `Cancel` calls GitHub's cancel-workflow-run endpoint with that run ID. | ||
|
|
||
| ## Minimal workflow | ||
|
|
||
| Create a workflow on the target repository's default branch: | ||
|
|
||
| ```yaml | ||
| name: SubmitQueue CI | ||
| run-name: SubmitQueue ${{ inputs.sq_queue }} | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| sq_base_uris: | ||
| required: true | ||
| type: string | ||
| sq_head_uris: | ||
| required: true | ||
| type: string | ||
| sq_queue: | ||
| required: true | ||
| type: string | ||
| sq_metadata: | ||
| required: false | ||
| type: string | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Inspect SubmitQueue payload | ||
| run: | | ||
| echo '${{ inputs.sq_base_uris }}' | ||
| echo '${{ inputs.sq_head_uris }}' | ||
| echo '${{ inputs.sq_queue }}' | ||
| # Prototype: add a script here that applies sq_base_uris, then | ||
| # sq_head_uris, then runs the repository's real CI command. | ||
| ``` | ||
|
|
||
| The workflow definition should live on a trusted ref. The untrusted changes | ||
| should be represented by `sq_base_uris` and `sq_head_uris` and applied inside | ||
| the job. | ||
|
|
||
| ## Integrator configuration | ||
|
|
||
| A server wiring this backend should provide the GitHub API client and runner | ||
| configuration equivalent to: | ||
|
|
||
| ```sh | ||
| BUILD_RUNNER=githubactions | ||
| GITHUB_BASE_URL=https://api.github.com | ||
| GITHUB_TOKEN=<token with actions:read/actions:write> | ||
| GITHUB_ACTIONS_OWNER=uber | ||
| GITHUB_ACTIONS_REPO=submitqueue | ||
| GITHUB_ACTIONS_WORKFLOW=submitqueue-ci.yml | ||
| GITHUB_ACTIONS_REF=main | ||
| GITHUB_ACTIONS_EXTRA_INPUTS=runner=ubuntu-latest,test_command=make test | ||
| ``` | ||
|
|
||
| `GITHUB_ACTIONS_REF` defaults to `main`. `GITHUB_ACTIONS_EXTRA_INPUTS` is | ||
| optional comma-separated `key=value` data copied into every dispatch request; | ||
| use it for workflow-specific knobs like runner labels or test commands. | ||
|
|
||
| ## Practical setup notes | ||
|
|
||
| - Use a trusted workflow definition from the target repository's default branch. | ||
| - Keep workflow permissions minimal. The job may apply untrusted changes before | ||
| running tests, so avoid broad secrets in this workflow. | ||
| - The backend proves the BuildRunner architecture. It does not prescribe how | ||
| your workflow materializes `sq_base_uris` and `sq_head_uris`; wire that to the | ||
| repository's existing patch/PR application logic. | ||
178 changes: 178 additions & 0 deletions
178
submitqueue/extension/buildrunner/githubactions/client.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| // Copyright (c) 2025 Uber Technologies, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package githubactions | ||
|
|
||
| import ( | ||
| "bytes" | ||
| "context" | ||
| "encoding/json" | ||
| "fmt" | ||
| "io" | ||
| "net/http" | ||
| "net/url" | ||
| "strconv" | ||
| ) | ||
|
|
||
| const githubAPIVersion = "2026-03-10" | ||
|
|
||
| // client is a thin wrapper around the GitHub Actions REST endpoints that | ||
| // BuildRunner needs: workflow dispatch, get workflow run, and cancel run. | ||
| type client struct { | ||
| httpClient *http.Client | ||
| owner string | ||
| repo string | ||
| workflowID string | ||
| } | ||
|
|
||
| type dispatchWorkflowRequest struct { | ||
| Ref string `json:"ref"` | ||
| ReturnRunDetails bool `json:"return_run_details,omitempty"` | ||
| Inputs map[string]string `json:"inputs,omitempty"` | ||
| } | ||
|
|
||
| type dispatchWorkflowResponse struct { | ||
| WorkflowRunID int64 `json:"workflow_run_id"` | ||
| RunURL string `json:"run_url"` | ||
| HTMLURL string `json:"html_url"` | ||
| } | ||
|
|
||
| // workflowRun is the subset of a GitHub Actions workflow run object the | ||
| // runner needs. | ||
| type workflowRun struct { | ||
| ID int64 `json:"id"` | ||
| Name string `json:"name"` | ||
| DisplayTitle string `json:"display_title"` | ||
| Status string `json:"status"` | ||
| Conclusion string `json:"conclusion"` | ||
| HTMLURL string `json:"html_url"` | ||
| RunAttempt int `json:"run_attempt"` | ||
| Event string `json:"event"` | ||
| HeadBranch string `json:"head_branch"` | ||
| CreatedAt string `json:"created_at"` | ||
| } | ||
|
|
||
| func (c *client) dispatchWorkflow(ctx context.Context, req dispatchWorkflowRequest) (dispatchWorkflowResponse, error) { | ||
| body, err := json.Marshal(req) | ||
| if err != nil { | ||
| return dispatchWorkflowResponse{}, fmt.Errorf("marshal request: %w", err) | ||
| } | ||
| var resp dispatchWorkflowResponse | ||
| if err := c.do(ctx, http.MethodPost, c.workflowPath("dispatches"), body, &resp); err != nil { | ||
| return dispatchWorkflowResponse{}, err | ||
| } | ||
| return resp, nil | ||
| } | ||
|
|
||
| func (c *client) getRun(ctx context.Context, runID int64) (workflowRun, error) { | ||
| var run workflowRun | ||
| if err := c.do(ctx, http.MethodGet, c.runPath(runID), nil, &run); err != nil { | ||
| return workflowRun{}, err | ||
| } | ||
| return run, nil | ||
| } | ||
|
|
||
| func (c *client) cancelRun(ctx context.Context, runID int64) error { | ||
| req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.runPath(runID)+"/cancel", nil) | ||
| if err != nil { | ||
| return fmt.Errorf("create request: %w", err) | ||
| } | ||
| c.setHeaders(req) | ||
|
|
||
| resp, err := c.httpClient.Do(req) | ||
| if err != nil { | ||
| return fmt.Errorf("send request: %w", err) | ||
| } | ||
| defer resp.Body.Close() | ||
| _, _ = io.Copy(io.Discard, resp.Body) | ||
|
|
||
| switch resp.StatusCode { | ||
| case http.StatusAccepted, http.StatusOK, http.StatusCreated, http.StatusNoContent: | ||
| return nil | ||
| case http.StatusConflict, http.StatusUnprocessableEntity: | ||
| // Already terminal or otherwise not cancellable: no-op per | ||
| // BuildRunner.Cancel contract. | ||
| return nil | ||
| case http.StatusNotFound: | ||
| return fmt.Errorf("workflow run not found") | ||
| default: | ||
| return fmt.Errorf("unexpected status %d from cancel", resp.StatusCode) | ||
|
albertywu marked this conversation as resolved.
|
||
| } | ||
|
albertywu marked this conversation as resolved.
albertywu marked this conversation as resolved.
|
||
| } | ||
|
albertywu marked this conversation as resolved.
|
||
|
|
||
| func (c *client) workflowPath(suffix string) string { | ||
| path := fmt.Sprintf( | ||
| "/repos/%s/%s/actions/workflows/%s", | ||
| url.PathEscape(c.owner), | ||
| url.PathEscape(c.repo), | ||
| url.PathEscape(c.workflowID), | ||
| ) | ||
| if suffix != "" { | ||
| path += "/" + suffix | ||
| } | ||
| return path | ||
| } | ||
|
|
||
| func (c *client) runPath(runID int64) string { | ||
| return fmt.Sprintf( | ||
| "/repos/%s/%s/actions/runs/%s", | ||
| url.PathEscape(c.owner), | ||
| url.PathEscape(c.repo), | ||
| url.PathEscape(strconv.FormatInt(runID, 10)), | ||
| ) | ||
| } | ||
|
|
||
| func (c *client) do(ctx context.Context, method, rawURL string, body []byte, out any) error { | ||
| var bodyReader io.Reader | ||
| if body != nil { | ||
| bodyReader = bytes.NewReader(body) | ||
| } | ||
|
|
||
| req, err := http.NewRequestWithContext(ctx, method, rawURL, bodyReader) | ||
| if err != nil { | ||
| return fmt.Errorf("create request: %w", err) | ||
| } | ||
| c.setHeaders(req) | ||
|
|
||
| resp, err := c.httpClient.Do(req) | ||
| if err != nil { | ||
| return fmt.Errorf("send request: %w", err) | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| respBody, err := io.ReadAll(resp.Body) | ||
| if err != nil { | ||
| return fmt.Errorf("read response: %w", err) | ||
| } | ||
|
|
||
| if resp.StatusCode == http.StatusNotFound { | ||
| return fmt.Errorf("GitHub API returned 404 for %s %s", method, rawURL) | ||
| } | ||
|
albertywu marked this conversation as resolved.
|
||
| if resp.StatusCode < 200 || resp.StatusCode >= 300 { | ||
| return fmt.Errorf("API returned status %d: %s", resp.StatusCode, respBody) | ||
| } | ||
|
|
||
| if out != nil && len(respBody) > 0 { | ||
| if err := json.Unmarshal(respBody, out); err != nil { | ||
| return fmt.Errorf("unmarshal response: %w", err) | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| func (c *client) setHeaders(req *http.Request) { | ||
| req.Header.Set("Accept", "application/vnd.github+json") | ||
| req.Header.Set("Content-Type", "application/json") | ||
| req.Header.Set("X-GitHub-Api-Version", githubAPIVersion) | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.