Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions .github/docs/NIGHTLY-BUILD-QUICK-REF.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,15 @@ Go to: **Actions** → **Nightly Build & Test** → **Run workflow**
| `test-backend-<branch>` | Backend tests + coverage against `<branch>` |
| `test-frontend-<branch>` | Frontend tests + coverage against `<branch>` |
| `deploy-<branch>` | Full stack deploy from `<branch>` + smoke test + teardown |
| `merge-validation:<base>:<overlay>` | Deploy `<base>`, overlay `<overlay>`, teardown |
| `all` | All of the above with defaults (`develop` for tests/deploy, `main`→`develop` for MV) |
| `e2e-<branch>` | Full stack deploy from `<branch>` + Playwright E2E tests + teardown |
| `scan-images-<branch>` | Build + Trivy-scan Docker images from `<branch>` |
| `all` | All of the above with defaults (`develop` for tests/deploy/e2e) |

### Examples
```
test-backend-develop
deploy-main,test-frontend-main
merge-validation:main:feature/my-branch
e2e-develop
all
```

Expand All @@ -42,13 +43,12 @@ Set `NIGHTLY_TRACKS` in **Settings → Secrets and variables → Actions → Var
### Test Tracks
1. ✅ Install dependencies + run tests with coverage
2. 📊 Compare coverage against previous baseline
3. 🤖 AI analysis creates GitHub issues for coverage gaps (`test-coverage` + `nightly-build` labels)

### Deploy Track
Full pipeline: infrarag → inferenceapp → frontend + gateway → smoke test → teardown
Full pipeline: platformbackend code deploys → frontend → smoke test → teardown

### Merge Validation
Deploys base branch, then overlays another branch on top — catches CDK/infra incompatibilities before merging.
### E2E Track
Deploys a full stack and runs Playwright E2E tests against it, then tears down.

## Debugging Failed Runs

Expand All @@ -57,7 +57,6 @@ Deploys base branch, then overlays another branch on top — catches CDK/infra i
| Nothing runs on schedule | Set `NIGHTLY_TRACKS` repo variable |
| Deploy fails | Check AWS credentials + CDK variables in development environment |
| Teardown fails | Manually empty S3 buckets + `npx cdk destroy --all --force` |
| MV overlay fails | Intended — overlay branch has infra incompatibilities with base |
| Coverage analysis fails | Check that test jobs uploaded artifacts |

## Cost Considerations
Expand Down
32 changes: 6 additions & 26 deletions .github/docs/NIGHTLY-BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

The nightly workflow validates the AgentCore Public Stack through configurable tracks — backend tests, frontend tests, full-stack deploys, and merge validation. It runs every night at 2 AM Mountain Time (9 AM UTC) and can be triggered manually.
The nightly workflow validates the AgentCore Public Stack through configurable tracks — backend tests, frontend tests, full-stack deploys, E2E tests, and image scanning. It runs every night at 2 AM Mountain Time (9 AM UTC) and can be triggered manually.

**Fork safety**: If the `NIGHTLY_TRACKS` repository variable is not set, no tracks run. Forked repos are safe by default.

Expand All @@ -18,8 +18,8 @@ Tracks are specified as a comma-separated string in the `NIGHTLY_TRACKS` repo va
| `test-frontend-<branch>` | Run frontend tests against `<branch>` |
| `deploy-<branch>` | Deploy full stack from `<branch>` with automatic teardown |
| `e2e-<branch>` | Deploy full stack from `<branch>`, run Playwright E2E tests, then teardown |
| `merge-validation:<base>:<overlay>` | Deploy `<base>`, then overlay `<overlay>` on top (colons delimit to avoid branch name ambiguity) |
| `all` | Run all tracks with defaults: tests + deploy + e2e on `develop`, MV `main`→`develop` |
| `scan-images-<branch>` | Build Docker images from `<branch>` and scan them with Trivy |
| `all` | Run all tracks with defaults: tests + deploy + e2e on `develop` |

### Examples

Expand All @@ -29,9 +29,7 @@ test-frontend-main,test-backend-main
deploy-develop
deploy-main,deploy-develop
e2e-develop
merge-validation:main:develop
merge-validation:main:feature/my-branch
test-backend-develop,deploy-develop,e2e-develop,merge-validation:main:develop
test-backend-develop,deploy-develop,e2e-develop
all
```

Expand All @@ -46,7 +44,7 @@ The `resolve-tracks` job parses the tracks string into boolean flags and branch
## Workflow Jobs

### resolve-tracks
Parses the tracks string and outputs boolean flags (`run_test_backend`, `run_test_frontend`, `run_deploy`, `run_e2e`, `run_mv`) and branch refs for each enabled track.
Parses the tracks string and outputs boolean flags (`run_test_backend`, `run_test_frontend`, `run_deploy`, `run_e2e`, `run_scan_images`) and branch refs for each enabled track.

### Test Tracks

Expand All @@ -55,7 +53,6 @@ When `test-backend-<branch>` or `test-frontend-<branch>` is specified:
1. **install-backend / install-frontend**: Install and cache dependencies
2. **test-backend / test-frontend**: Run test suites with coverage, upload artifacts
3. **analyze-coverage**: Compare coverage against previous baseline (runs if any test succeeded)
4. **ai-coverage-analysis**: Uses GitHub Models API (GPT-4o) to analyze coverage gaps and create/update GitHub issues labeled `test-coverage` + `nightly-build`

### Deploy Track

Expand All @@ -82,15 +79,6 @@ E2E test failures are **informational** — they mark the nightly summary as "pa
- `E2E_ADMIN_USERNAME` / `E2E_ADMIN_PASSWORD` — Cognito admin test account
- `E2E_USER_USERNAME` / `E2E_USER_PASSWORD` — Cognito regular user test account

### Merge Validation Track

When `merge-validation:<base>:<overlay>` is specified:

1. **mv-base**: Deploys `<base>` branch with `nightly-mv` prefix, `skip-teardown: true`. Uses `source-project-prefix: dev-boisestateai-v2` for Docker image promotion (promote-or-build pattern).
2. **mv-overlay**: Deploys `<overlay>` branch on top of the same `nightly-mv` stack, then tears down.

This simulates a real merge to catch CDK/infra incompatibilities between branches.

### Summary

Generates a GitHub Actions job summary table showing the status of all enabled tracks.
Expand All @@ -108,13 +96,8 @@ Reusable workflow (`workflow_call`) containing the full deploy pipeline.
| `alb-subdomain` | yes | ALB subdomain for the deployment |
| `skip-teardown` | no | Skip teardown (default: `false`) |
| `label` | no | Label for job names |
| `source-project-prefix` | no | If set, Docker jobs try ECR image promotion before building |
| `run-e2e` | no | Run Playwright E2E tests after smoke test (default: `false`) |

### Promote-or-Build Pattern

When `source-project-prefix` is provided, Docker jobs (rag-ingestion, inference-api, app-api) attempt to promote existing images from the source ECR before falling back to a full build. This avoids unnecessary Docker builds when images haven't changed.

## Manual Triggers

Go to: **Actions** → **Nightly Build & Test** → **Run workflow**
Expand All @@ -138,7 +121,7 @@ Example values:

### Environment

Deploy and MV tracks use the `development` GitHub environment with overrides:
Deploy and E2E tracks use the `development` GitHub environment with overrides:
- `CDK_RETAIN_DATA_ON_DELETE=false` (enables clean teardown)
- Minimal resource sizing

Expand Down Expand Up @@ -179,9 +162,6 @@ Check that `NIGHTLY_TRACKS` is set as a repository variable (not a secret). Empt
- Ensure test jobs uploaded coverage artifacts
- Check Python script logs for errors

### Merge validation fails on overlay
This is the intended signal — it means the overlay branch has CDK/infra incompatibilities with the base branch that need to be resolved before merging.

### E2E tests fail
- Download the `playwright-report-<label>` artifact for screenshots and traces
- Check that `E2E_ADMIN_USERNAME`, `E2E_ADMIN_PASSWORD`, `E2E_USER_USERNAME`, `E2E_USER_PASSWORD` secrets are set in the `development` environment
Expand Down
56 changes: 17 additions & 39 deletions .github/workflows/backend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,13 @@ name: Backend Deploy

on:
workflow_dispatch:
# Push-triggered deploys are disabled for the v1.0.0 release so that
# forking/syncing the codebase never auto-deploys backend code into a
# user's AWS account. Deploy intentionally via the Actions tab. Re-enable
# by uncommenting the block below.
# push:
# branches: [develop, main]
# paths:
# - 'backend/**'
# - 'scripts/build/**'
# - '.github/workflows/backend.yml'
# Deploy on pushes to backend code or its build plumbing.
push:
branches: [develop, main]
paths:
- 'backend/**'
- 'scripts/build/**'
- '.github/workflows/backend.yml'

permissions:
contents: read
Expand All @@ -20,8 +17,11 @@ env:
CDK_REQUIRE_APPROVAL: never
LOAD_ENV_QUIET: true

# Shared "deploy-<ref>" group with platform.yml so a CFN deploy and these
# API-driven code deploys never mutate the same ECS service / AgentCore
# Runtime / Lambda concurrently — they queue instead.
concurrency:
group: backend-${{ github.ref }}
group: deploy-${{ github.ref }}
cancel-in-progress: false

# ------------------------------------------------------------
Expand All @@ -42,41 +42,19 @@ concurrency:
jobs:
test-infra:
name: Test infrastructure (jest)
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: infrastructure/package-lock.json
- name: Install
working-directory: infrastructure
run: npm ci --prefer-offline
- name: Run tests
working-directory: infrastructure
run: npx jest --maxWorkers=2
uses: ./.github/workflows/tests.yml
with:
run_infra: true

test-backend:
name: Test backend (pytest)
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: '0.7.12'
enable-cache: true
cache-dependency-glob: 'backend/uv.lock'
- name: Sync deps
working-directory: backend
run: uv sync --extra agentcore --extra dev
- name: Run pytest
working-directory: backend
run: uv run pytest tests/ -v
uses: ./.github/workflows/tests.yml
with:
run_backend: true

build-app-api:
name: Build app-api image
Expand Down
60 changes: 7 additions & 53 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,59 +23,13 @@ concurrency:
cancel-in-progress: true

jobs:
test-backend:
name: Test backend (pytest)
runs-on: ubuntu-24.04
tests:
name: Tests
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: '0.7.12'
enable-cache: true
cache-dependency-glob: 'backend/uv.lock'
- name: Sync deps
working-directory: backend
run: uv sync --extra agentcore --extra dev
- name: Run pytest
working-directory: backend
run: uv run pytest tests/ -v
uses: ./.github/workflows/tests.yml
with:
run_backend: true
run_frontend: true
run_infra: true

test-frontend:
name: Test frontend (vitest)
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: frontend/ai.client/package-lock.json
- name: Install
working-directory: frontend/ai.client
run: npm ci --prefer-offline
- name: Run tests
working-directory: frontend/ai.client
run: npm run test:ci

test-infra:
name: Test infrastructure (jest)
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: infrastructure/package-lock.json
- name: Install
working-directory: infrastructure
run: npm ci --prefer-offline
- name: Run tests
working-directory: infrastructure
run: npx jest --maxWorkers=2
6 changes: 5 additions & 1 deletion .github/workflows/docs-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ name: Docs Deploy
on:
workflow_dispatch:
push:
branches: [develop, main]
# The github-pages environment must allow main to deploy.
branches: [main]
paths:
- 'docs-site/**'
- '.github/workflows/docs-deploy.yml'
Expand All @@ -20,6 +21,8 @@ concurrency:
jobs:
build:
name: Build Starlight site
# Don't auto-publish from forks (manual dispatch still works).
if: ${{ github.repository == 'Boise-State-Development/agentcore-public-stack' || github.event_name == 'workflow_dispatch' }}
runs-on: ubuntu-24.04
permissions:
contents: read
Expand All @@ -44,6 +47,7 @@ jobs:
deploy:
name: Deploy to GitHub Pages
needs: build
if: ${{ github.repository == 'Boise-State-Development/agentcore-public-stack' || github.event_name == 'workflow_dispatch' }}
runs-on: ubuntu-24.04
permissions:
pages: write
Expand Down
33 changes: 10 additions & 23 deletions .github/workflows/frontend-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,13 @@ name: Frontend Deploy

on:
workflow_dispatch:
# Push-triggered deploys are disabled for the v1.0.0 release so that
# forking/syncing the codebase never auto-deploys the SPA into a user's
# AWS account. Deploy intentionally via the Actions tab. Re-enable by
# uncommenting the block below.
# push:
# branches: [develop, main]
# paths:
# - 'frontend/**'
# - 'scripts/frontend/**'
# Deploy on pushes to the SPA or its deploy plumbing.
push:
branches: [develop, main]
paths:
- 'frontend/**'
- 'scripts/frontend/**'
- '.github/workflows/frontend-deploy.yml'

permissions:
contents: read
Expand All @@ -25,22 +23,11 @@ concurrency:
jobs:
test-frontend:
name: Test frontend (ng test)
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: frontend/ai.client/package-lock.json
- name: Install
working-directory: frontend/ai.client
run: npm ci --prefer-offline
- name: Run tests
working-directory: frontend/ai.client
run: npm run test:ci
uses: ./.github/workflows/tests.yml
with:
run_frontend: true

build:
name: Build SPA
Expand Down
Loading