forked from github-community-projects/safe-settings
-
Notifications
You must be signed in to change notification settings - Fork 0
ci: automate safe-settings deployment #17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,70 +1,111 @@ | ||
| name: Deploy to Kubernetes | ||
| on: | ||
| name: Deploy safe-settings | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main-enterprise | ||
| paths: | ||
| - .github/workflows/deploy-k8s.yml | ||
| - Dockerfile | ||
| - index.js | ||
| - lib/** | ||
| - package.json | ||
| - package-lock.json | ||
| - helm/safe-settings/** | ||
| workflow_dispatch: | ||
| inputs: | ||
| status: | ||
| description: 'Status of the previous workflow' | ||
| required: true | ||
| default: 'passed' | ||
| release: | ||
| description: 'Release tag from the previous workflow' | ||
| required: true | ||
| env: | ||
| DOCKER_IMAGE_NAME: yadhav/safe-settings | ||
| #IMAGE_REGISTRY_URL: docker.pkg.github.com | ||
| IMAGE_REGISTRY_URL: docker.io | ||
| AZURE_RESOURCE_GROUP: decyjphr-aks | ||
| AZURE_AKS_CLUSTER: decyjphr-aks | ||
| AZURE_LOCATION: '"East US"' | ||
| AZURE_AKS_NAMESPACE: default | ||
|
|
||
| permissions: | ||
| id-token: write | ||
|
|
||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: safe-settings-production | ||
| cancel-in-progress: false | ||
|
|
||
| env: | ||
| GCP_PROJECT_ID: hacktron-462816 | ||
| GCP_REGION: us-central1 | ||
| ARTIFACT_REPOSITORY: safe-settings | ||
| IMAGE_NAME: github-app | ||
| GKE_CLUSTER: safe-settings-cluster | ||
| GKE_LOCATION: us-central1-a | ||
| K8S_NAMESPACE: safe-settings | ||
| HELM_RELEASE: safe-settings | ||
| HELM_DRIVER: configmap | ||
|
|
||
| jobs: | ||
| deploy-to-k8s: | ||
| if: ${{ github.event.inputs.status == 'passed' }} | ||
| name: Deploy to Kubernetes | ||
| test: | ||
| name: Test | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v6 | ||
| - uses: azure/login@a457da9ea143d694b1b9c7c869ebb04ebe844ef5 | ||
| with: | ||
| client-id: ${{ secrets.AZURE_CLIENT_ID }} | ||
| tenant-id: ${{ secrets.AZURE_TENANT_ID }} | ||
| subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} | ||
| - uses: azure/aks-set-context@c7eb093e5a5d47caa333f64974d5fd1cd4bf069d | ||
| - uses: actions/checkout@v7 | ||
| - uses: actions/setup-node@v7 | ||
| with: | ||
| resource-group: ${{env.AZURE_RESOURCE_GROUP}} | ||
| cluster-name: ${{env.AZURE_AKS_CLUSTER}} | ||
| id: login | ||
| - run: | | ||
| kubectl get deployment | ||
| - name: app-env | ||
| uses: azure/k8s-create-secret@6e0ba8047235646753f2a3a3b359b4d0006ff218 | ||
| node-version: 24 | ||
| cache: npm | ||
| - run: npm ci | ||
| - run: npm run test:unit:ci | ||
|
|
||
| deploy: | ||
| name: Deploy production | ||
| if: ${{ vars.SAFE_SETTINGS_DEPLOY_ENABLED == 'true' }} | ||
| needs: test | ||
| runs-on: ubuntu-latest | ||
| environment: production | ||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - id: auth | ||
| name: Authenticate to Google Cloud | ||
| uses: google-github-actions/auth@v3 | ||
| with: | ||
| namespace: 'default' | ||
| secret-type: 'generic' | ||
| arguments: --from-literal=APP_ID=${{ secrets.APP_ID }} --from-literal=PRIVATE_KEY=${{ secrets.PRIVATE_KEY }} --from-literal=WEBHOOK_SECRET=${{ secrets.WEBHOOK_SECRET }} | ||
| secret-name: app-env | ||
| - name: Set imagePullSecret | ||
| uses: azure/k8s-create-secret@6e0ba8047235646753f2a3a3b359b4d0006ff218 | ||
| workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }} | ||
| service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }} | ||
|
|
||
| - name: Set up Google Cloud SDK | ||
| uses: google-github-actions/setup-gcloud@v3 | ||
|
|
||
| - name: Configure Docker for Artifact Registry | ||
| run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v4 | ||
|
|
||
| - name: Build and push image | ||
| uses: docker/build-push-action@v7 | ||
| with: | ||
| namespace: ${{env.AZURE_AKS_NAMESPACE}} | ||
| container-registry-url: ${{env.IMAGE_REGISTRY_URL}} | ||
| container-registry-username: ${{ secrets.DOCKER_USERNAME }} | ||
| container-registry-password: ${{ secrets.DOCKER_PASSWORD }} | ||
| secret-name: 'image-pull-secret' | ||
| id: create-secret | ||
| - uses: Azure/k8s-deploy@v5 | ||
| context: . | ||
| push: true | ||
| tags: ${{ env.GCP_REGION }}-docker.pkg.dev/${{ env.GCP_PROJECT_ID }}/${{ env.ARTIFACT_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} | ||
| cache-from: type=gha | ||
| cache-to: type=gha,mode=max,ignore-error=true | ||
|
|
||
| - name: Authenticate to GKE | ||
| uses: google-github-actions/get-gke-credentials@v3 | ||
| with: | ||
| namespace: ${{env.AZURE_AKS_NAMESPACE}} | ||
| manifests: | | ||
| safe-settings.yaml | ||
| svc-safe-settings.yaml | ||
| #images: ${{env.IMAGE_REGISTRY_URL}}/${{ github.repository }}/${{env.DOCKER_IMAGE_NAME}}:${{ github.sha }} | ||
| images: ${{env.IMAGE_REGISTRY_URL}}/yadhav/safe-settings:${{ github.event.inputs.release }} | ||
| imagepullsecrets: | | ||
| image-pull-secret | ||
| kubectl-version: 'latest' | ||
| cluster_name: ${{ env.GKE_CLUSTER }} | ||
| location: ${{ env.GKE_LOCATION }} | ||
|
|
||
| - name: Deploy with Helm | ||
| run: | | ||
| helm upgrade --install "${HELM_RELEASE}" ./helm/safe-settings \ | ||
| --namespace "${K8S_NAMESPACE}" \ | ||
| --values helm/safe-settings/values-hacktron.yaml \ | ||
| --set-string image.tag="${GITHUB_SHA}" \ | ||
| --atomic \ | ||
| --wait \ | ||
| --timeout 10m \ | ||
| --history-max 10 | ||
|
|
||
| - name: Verify rollout | ||
| run: kubectl --namespace "${K8S_NAMESPACE}" rollout status deployment/"${HELM_RELEASE}" --timeout=5m | ||
|
|
||
| - name: Record deployment | ||
| run: | | ||
| { | ||
| echo "### safe-settings deployed" | ||
| echo "- Image: \`${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${ARTIFACT_REPOSITORY}/${IMAGE_NAME}:${GITHUB_SHA}\`" | ||
| echo "- Cluster: \`${GKE_CLUSTER}\` (\`${GKE_LOCATION}\`)" | ||
| } >> "${GITHUB_STEP_SUMMARY}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,60 +1,132 @@ | ||
| # Deployment | ||
|
|
||
| This is our own documentation for deploying to GCP. In general this should only need to be done once, unless we need to modify any of the core logic. Changes to settings in [`.github`](./.github) will be automatically picked up. | ||
| Safe Settings is deployed to the existing GKE target by | ||
| `.github/workflows/deploy-k8s.yml`. A push to `main-enterprise` automatically | ||
| deploys application, image, chart, or workflow changes after unit tests pass. | ||
| Changes limited to `.github/repos/*.yml` do not rebuild the application. | ||
|
|
||
| Authenticate: | ||
| Deployment is deliberately disabled until the repository variable | ||
| `SAFE_SETTINGS_DEPLOY_ENABLED` is set to `true`. | ||
|
|
||
| ```bash | ||
| $ gcloud auth print-access-token | helm registry login -u oauth2accesstoken \ | ||
| --password-stdin us-central1-docker.pkg.dev | ||
| ``` | ||
| ## One-time GitHub Actions setup | ||
|
|
||
| Create a `production` environment and make these secrets available to the | ||
| repository, either by sharing the existing Hacktron organization secrets or by | ||
| adding repository/environment secrets with the same names: | ||
|
|
||
| - `GCP_WORKLOAD_IDENTITY_PROVIDER` | ||
| - `GCP_SERVICE_ACCOUNT` | ||
|
|
||
| The Workload Identity provider must map `attribute.repository` and trust | ||
| `HacktronAI/safe-settings`. Use the dedicated service account | ||
| `safe-settings-deployer@hacktron-462816.iam.gserviceaccount.com`; do not reuse a | ||
| broad infrastructure runner. It needs: | ||
|
|
||
| - `roles/artifactregistry.writer` on only the `safe-settings` Artifact Registry | ||
| repository | ||
| - `roles/container.clusterViewer` on project `hacktron-462816` | ||
| - `roles/iam.workloadIdentityUser` granted to the GitHub OIDC principal on the | ||
| service account | ||
|
|
||
| Package Helm chart: | ||
| Create the dedicated namespace and apply its scoped Kubernetes role once from | ||
| an administrator context: | ||
|
|
||
| ```bash | ||
| $ helm package helm/safe-settings | ||
| Successfully packaged chart and saved it to: /home/zeyu/OneDrive/Documents/Projects/hacktron/safe-settings/safe-settings-0.1.0.tgz | ||
| kubectl apply --filename helm/safe-settings/namespace.yaml | ||
| kubectl apply --filename helm/safe-settings/deployer-rbac.yaml | ||
| ``` | ||
|
|
||
| Push image to GCP artifact repository: | ||
| This limits the CI identity to the Helm resources used by this chart in the | ||
| `safe-settings` namespace instead of granting project-wide Kubernetes | ||
| developer access. Helm stores its release metadata in ConfigMaps so CI does not | ||
| need permission to read Kubernetes Secrets. | ||
|
|
||
| ```bash | ||
| $ gcloud artifacts repositories create safe-settings \ | ||
| --repository-format=docker \ | ||
| --location=us-central1 \ | ||
| --description="GitHub policy-as-code" | ||
| Create request issued for: [safe-settings] | ||
| Waiting for operation [projects/hacktron-462816/locations/us-central1/operations/b15dad24-1d36-409d-8d81-64f983554982] to complete...done. | ||
| Created repository [safe-settings]. | ||
| The workflow currently targets: | ||
|
|
||
| - project: `hacktron-462816` | ||
| - registry: `us-central1-docker.pkg.dev/hacktron-462816/safe-settings` | ||
| - cluster: `safe-settings-cluster` | ||
| - cluster location: `us-central1-a` | ||
| - Helm release and deployment: `safe-settings` | ||
| - namespace: `safe-settings` | ||
|
|
||
| $ docker tag safe-settings us-central1-docker.pkg.dev/hacktron-462816/safe-settings/github-app:0.1.0 | ||
| Only enable deployment after those resources and permissions are confirmed: | ||
|
|
||
| $ docker push us-central1-docker.pkg.dev/hacktron-462816/safe-settings/github-app:0.1.0 | ||
| ```bash | ||
| gh variable set SAFE_SETTINGS_DEPLOY_ENABLED \ | ||
| --repo HacktronAI/safe-settings \ | ||
| --body true | ||
| ``` | ||
|
|
||
| Push Helm chart: | ||
| ## One-time runtime secret setup | ||
|
|
||
| The workflow never copies GitHub App credentials into an image or command | ||
| line. The pod reads them from the `safe-settings/app-env` Kubernetes Secret. | ||
| The required keys are: | ||
|
|
||
| - `APP_ID` | ||
| - `PRIVATE_KEY` | ||
| - `WEBHOOK_SECRET` | ||
| - `WEBHOOK_PROXY_URL` while Smee is used | ||
|
|
||
| After authenticating to GCP and selecting the cluster, populate the Secret from | ||
| the ignored local `.env` file: | ||
|
|
||
| ```bash | ||
| $ helm push safe-settings-0.1.0.tgz oci://us-central1-docker.pkg.dev/hacktron-462816/safe-settings | ||
| Pushed: us-central1-docker.pkg.dev/hacktron-462816/safe-settings/safe-settings:0.1.0 | ||
| Digest: sha256:818ad22a4f3fec92dbfcf0d244e35024cc25db35a88584742b92a31c00c37656 | ||
| gcloud auth login | ||
| gcloud container clusters get-credentials safe-settings-cluster \ | ||
| --project hacktron-462816 \ | ||
| --zone us-central1-a | ||
| kubectl apply --filename helm/safe-settings/namespace.yaml | ||
| K8S_NAMESPACE=safe-settings ./script/bootstrap-k8s-secret | ||
| ``` | ||
|
|
||
| Deploy Helm chart: | ||
| The bootstrap script writes values only to a private temporary directory, | ||
| applies the Secret, and removes the temporary files without printing values. | ||
|
|
||
| ```bash | ||
| $ gcloud container clusters create --zone us-central1-a safe-settings-cluster | ||
| $ gcloud container clusters get-credentials --zone us-central1-a safe-settings-cluster | ||
| ## Webhooks | ||
|
|
||
| $ source .env | ||
| helm install safe-settings oci://us-central1-docker.pkg.dev/hacktron-462816/safe-settings/safe-settings --version 0.1.0 --set env.ADMIN_REPO="$ADMIN_REPO" --set env.GH_ORG="$GH_ORG" --set env.CRON="$CRON" --set env.APP_ID="\"$APP_ID\"" --set env.PRIVATE_KEY="$PRIVATE_KEY" --set env.WEBHOOK_SECRET="$WEBHOOK_SECRET" --set env.GITHUB_CLIENT_ID="$GITHUB_CLIENT_ID" --set env.GITHUB_CLIENT_SECRET="$GITHUB_CLIENT_SECRET" --set env.WEBHOOK_PROXY_URL="$WEBHOOK_PROXY_URL" | ||
| The GitHub App must have an active runtime and a webhook transport. The current | ||
| App configuration points to Smee. Keeping `WEBHOOK_PROXY_URL` in `app-env` | ||
| causes Probot to connect outbound to that Smee channel; a public Kubernetes | ||
| Ingress is not required for this initial setup. The webhook endpoint inside the | ||
| application is `/api/github/webhooks`. | ||
|
|
||
| $ helm upgrade --install safe-settings oci://us-central1-docker.pkg.dev/hacktron-462816/safe-settings/safe-settings --set env.ADMIN_REPO="$ADMIN_REPO" --set env.GH_ORG="$GH_ORG" --set env.CRON="$CRON" --set env.APP_ID="$APP_ID" --set env.PRIVATE_KEY="$PRIVATE_KEY" --set env.WEBHOOK_SECRET="$WEBHOOK_SECRET" --set env.GITHUB_CLIENT_ID="$GITHUB_CLIENT_ID" --set env.GITHUB_CLIENT_SECRET="$GITHUB_CLIENT_SECRET" --set env.WEBHOOK_PROXY_URL="$WEBHOOK_PROXY_URL" --set env.LOG_LEVEL="$LOG_LEVEL" --force | ||
| ``` | ||
| For a direct production webhook later: | ||
|
|
||
| If we need to SSH into the nodes through `gcloud computer ssh --tunnel-through-iap`: | ||
| 1. Provide a DNS hostname and HTTPS certificate. | ||
| 2. Enable the Helm ingress for that hostname. | ||
| 3. Change the GitHub App webhook URL to | ||
| `https://<hostname>/api/github/webhooks` with SSL verification enabled. | ||
| 4. Remove `WEBHOOK_PROXY_URL` from `.env` and run | ||
| `DIRECT_WEBHOOK=true ./script/bootstrap-k8s-secret`; the explicit flag | ||
| removes the proxy key from `app-env`. | ||
| 5. Send a test delivery and confirm a `2xx` response plus application logs for | ||
| that delivery. | ||
|
|
||
| ```bash | ||
| gcloud container node-pools update --zone us-central1-a default-pool --cluster=safe-settings-cluster --tags=ssh-iap | ||
| gcloud container clusters update --zone us-central1-a safe-settings-cluster --autoprovisioning-network-tags=ssh-iap | ||
| ``` | ||
| Smee returning `200` only confirms that Smee accepted a GitHub delivery; it | ||
| does not prove a Safe Settings pod was connected to consume it. | ||
|
|
||
| ## First deployment cutover | ||
|
|
||
| The legacy release runs as `default/safe-settings`. Do not run both releases | ||
| against the same Smee channel because a delivery could be processed twice. | ||
| After this workflow is merged, perform the first deployment as a short, | ||
| reversible cutover: | ||
|
|
||
| 1. Keep `SAFE_SETTINGS_DEPLOY_ENABLED=false` while merging the workflow. | ||
| 2. Scale `default/safe-settings` to zero replicas. | ||
| 3. Set `SAFE_SETTINGS_DEPLOY_ENABLED=true` and manually run the deployment | ||
| workflow from `main-enterprise`. | ||
| 4. Verify `safe-settings/safe-settings` is ready and its logs show the Smee | ||
| client receiving a delivery. | ||
| 5. If verification fails, set the variable back to `false` and scale the | ||
| legacy deployment back to one replica. | ||
|
|
||
| After a successful cutover, pushes to `main-enterprise` deploy automatically. | ||
|
|
||
| ## Manual deployment | ||
|
|
||
| The Actions workflow can also be started from **Actions → Deploy | ||
| safe-settings → Run workflow**. It uses the same tests, immutable image tag, | ||
| runtime Secret check, and atomic Helm deployment as automatic pushes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: Role | ||
| metadata: | ||
| name: safe-settings-deployer | ||
| namespace: safe-settings | ||
| rules: | ||
| - apiGroups: [""] | ||
| resources: | ||
| - configmaps | ||
| - serviceaccounts | ||
| - services | ||
| verbs: ["create", "delete", "get", "list", "patch", "update", "watch"] | ||
| - apiGroups: [""] | ||
| resources: | ||
| - endpoints | ||
| - pods | ||
| verbs: ["get", "list", "watch"] | ||
| - apiGroups: ["apps"] | ||
| resources: | ||
| - deployments | ||
| - replicasets | ||
| verbs: ["create", "delete", "get", "list", "patch", "update", "watch"] | ||
| - apiGroups: ["autoscaling"] | ||
| resources: | ||
| - horizontalpodautoscalers | ||
| verbs: ["create", "delete", "get", "list", "patch", "update", "watch"] | ||
| - apiGroups: ["networking.k8s.io"] | ||
| resources: | ||
| - ingresses | ||
| verbs: ["create", "delete", "get", "list", "patch", "update", "watch"] | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: RoleBinding | ||
| metadata: | ||
| name: safe-settings-deployer | ||
| namespace: safe-settings | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: Role | ||
| name: safe-settings-deployer | ||
| subjects: | ||
| - apiGroup: rbac.authorization.k8s.io | ||
| kind: User | ||
| name: safe-settings-deployer@hacktron-462816.iam.gserviceaccount.com |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: safe-settings |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When this workflow is manually dispatched from a feature branch or tag, the push branch filter does not apply and this condition only checks the repository variable, so the selected ref is checked out, built, and deployed to production. The local
gh workflow run --helpexplicitly documents--refas accepting a “Branch or tag name,” making it possible to bypassmain-enterprisereview and branch protection; add agithub.ref == 'refs/heads/main-enterprise'guard to the deploy job.Useful? React with 👍 / 👎.