Gate the seed on an SDK_LANGUAGES profile and check that its generate targets do work

.github/workflows/ci.yml

@@ -53,10 +53,11 @@ jobs:
           go-version: stable
           cache: false
 
-      - name: Install actionlint and zizmor
+      - name: Install actionlint, zizmor and bats
         run: |
           go install "github.com/rhysd/actionlint/cmd/actionlint@$ACTIONLINT_VERSION"
           pipx install "zizmor==$ZIZMOR_VERSION"
+          sudo apt-get install -y --no-install-recommends bats
 
       - name: Render seed/ with fixture values
         run: hack/render-seed.sh .rendered
@@ -74,6 +75,12 @@ jobs:
       - name: shellcheck (rendered scripts)
         run: find .rendered/scripts -type f -name '*.sh' -print0 | xargs -0 -r shellcheck
 
+      # The generate-target coverage check, pinned on its own renders: the pristine
+      # seed fails for all six languages, stubbed generators pass, a target that
+      # exits 0 having done nothing fails.
+      - name: bats (generate-target coverage check)
+        run: bats hack/test
+
   rust-scaffold:
     # seed/rust/ is the one scaffold that is a complete crate (AGENTS.md), so it is
     # the one that can be compiled before it merges: the rendered workspace must

AGENTS.md

@@ -69,13 +69,35 @@ inline, which is why the work sits under the unsuffixed name.
 `seed/Makefile.tmpl` defines the vocabulary. `actions/service-drift/action.yml`,
 `prompts/seed-sdk.md`, `prompts/close-gap.md`, `seed/AGENTS.md.tmpl`,
 `seed/CONTRIBUTING.md.tmpl`, `seed/README.md.tmpl`,
-`seed/.github/workflows/release-kotlin.yml.tmpl` and
-`seed/scripts/check-rust-service-drift.sh.tmpl` all restate it. Treat that list as a
-starting point and grep for the target name rather than trusting it. Rename a target and every one of them has to move in the same commit — they have
+`seed/.github/workflows/release-kotlin.yml.tmpl`,
+`seed/scripts/check-generate-targets.sh` (which also names the generator artifact each
+recipe invokes) and `seed/scripts/check-rust-service-drift.sh.tmpl` all restate it.
+Treat that list as a starting point and grep for the target name rather than trusting
+it. Rename a target and every one of them has to move in the same commit — they have
 drifted apart before. The Kotlin workflow is easy to miss: it names
 `kt-generate-services` inside a drift-check error message, so a stale name there ships
 into every SDK generated afterwards and tells users to run a target that is gone.
 
+## The language profile is `SDK_LANGUAGES`, and the generate targets are checked
+
+`SDK_LANGUAGES` at the top of `seed/Makefile.tmpl` is the only place a seeded SDK's
+language set lives; `check`, `generate-services`, `conformance` and `clean` range over
+it (`MAKEFILE-CONVENTION.md` § Language Profile). Adding a language to the seed means
+adding its prefix to that default, a `CONFORMANCE_RUNNER_<prefix>` mapping if its runner
+belongs in the aggregate, its generator artifact to `seed/scripts/check-generate-targets.sh`,
+and a matrix entry in `seed/.github/workflows/test.yml.tmpl`. Directory presence is not
+the signal: an aggregate that reads the tree would hide a scaffold step that failed.
+
+The seed ships no generator for any language, so on a pristine render every
+`<prefix>-generate-services` target fails, and nothing here can prove a seeded repo's
+generators work. What `hack/test/check-generate-targets.bats` pins instead is the
+detector: `scripts/check-generate-targets.sh` must fail on the pristine render for all
+six prefixes, pass once the artifacts each recipe invokes exist (Swift's through its
+sub-Makefile), and fail on a target that exits 0 having done nothing -- including one
+whose own recipe is gone while a prerequisite such as `ts-install` still plans work.
+Rename a generate target, move a generator or change how a recipe invokes it and that
+test is the first thing to break.
+
 ## `seed/rust/` is a crate, not a fragment
 
 The other language scaffolds are fragments that compile only once the instantiator has

MAKEFILE-CONVENTION.md

@@ -6,7 +6,7 @@ Required Makefile targets and release architecture for SDK repositories.
 
 | Target | Default? | Contract |
 |--------|----------|----------|
-| `check` | **Yes** | CI gate. Comprehensive: smithy-check + behavior-model-check + url-routes-check + provenance-check + sync-api-version-check + {lang}-check + conformance + audit-check. Not fast — use `check-mvp` or `{lang}-check` for inner-loop. |
+| `check` | **Yes** | CI gate. Comprehensive: smithy-check + behavior-model-check + url-routes-check + provenance-check + sync-api-version-check + audit-check + generate-services-check + {lang}-check + conformance, with {lang}-check and conformance ranging over `SDK_LANGUAGES` (see Language Profile). Not fast — use `check-mvp` or `{lang}-check` for inner-loop. |
 | `check-mvp` | | Fast iteration target: smithy-check + behavior-model-check + url-routes-check + sync-api-version-check + go-check. Skips conformance. |
 | `smithy-mapper` | | Build Smithy plugin to local Maven |
 | `smithy-build` | | Build OpenAPI from Smithy. Prerequisites: `behavior-model smithy-mapper`. Post-step: `sync-api-version`. |
@@ -23,11 +23,12 @@ Required Makefile targets and release architecture for SDK repositories.
 | `{lang}-check` | | All checks for one language. Must include lint + test (+ typecheck where applicable). |
 | `{lang}-check-drift` | | Verify generated services match spec for one language |
 | `{lang}-generate-services` | | Generate service classes from OpenAPI |
-| `conformance` | | All cross-language conformance tests |
+| `conformance` | | Conformance runners for the languages in `SDK_LANGUAGES` |
 | `audit-check` | | Validate rubric-audit.json: exists, must-pass manual criteria pass, date within 30 days |
 | `bump VERSION=x.y.z` | | Atomic version bump across all languages |
 | `release VERSION=x.y.z` | | Sole release authority (see below) |
-| `generate-services` | | Aggregate: regenerate services for all languages |
+| `generate-services` | | Aggregate: `{lang}-generate-services` for every language in `SDK_LANGUAGES` |
+| `generate-services-check` | | Non-mutating, part of `check`: every `{lang}-generate-services` in `SDK_LANGUAGES` names a generator artifact that exists and plans its invocation, through sub-Makefile delegation; a prerequisite's commands do not count (`scripts/check-generate-targets.sh --dry-run`) |
 
 ## Naming Conventions
 
@@ -58,6 +59,39 @@ Required Makefile targets and release architecture for SDK repositories.
 | Kotlin | test (via `./gradlew :{app}-sdk:check`) |
 | Rust | fmt-check + clippy + test + doc + deny |
 
+## Language Profile
+
+The languages an SDK ships are one variable at the top of the Makefile, by prefix:
+
+```makefile
+SDK_LANGUAGES := go ts rb swift kt rs
+```
+
+It is set once, at instantiation, to the profile chosen in `prompts/seed-sdk.md`; the
+full set is the seed's default. The aggregates derive from it and nothing else:
+
+- `check`, `check-full`: `{lang}-check` for each listed prefix
+- `check-mvp`: adds `go-check` only when `go` is listed
+- `provenance-check` (Go-embedded provenance): in `check`/`check-full` only when `go` is listed
+- `generate-services`, `generate-services-check`, `clean`: one target per listed prefix
+- `conformance`, `conformance-build`: the runners for the listed prefixes, mapped by
+  `CONFORMANCE_RUNNER_<prefix>` (`ts` → `conformance-typescript`, and so on). Swift has
+  no mapping: the aggregate does not run the Swift runner today; the seeded `test.yml`
+  runs it on a macOS matrix leg instead
+
+Two things the profile deliberately does not do. It does not remove or guard the
+per-language targets: `make ts-check` on a profile without `ts` still exists and still
+fails, loudly, on the missing tree. And it is not inferred from which directories exist:
+a scaffold step that silently failed to create `go/` must fail `go-check`, not quietly
+narrow the gate. `sync-api-version-check` and `scripts/sync-version.sh` are the one
+exception, skipping a language whose file is absent, because a version constant that
+does not exist has nothing to be out of sync with.
+
+Instantiating a narrower profile therefore means: set `SDK_LANGUAGES`, and trim the
+`conformance` matrix in `.github/workflows/test.yml` to the same languages. `make
+release`'s version-constant guards still name all six languages and are not yet
+profile-aware.
+
 ## Release Architecture
 
 One path, no ambiguity:

hack/test/check-generate-targets.bats

@@ -0,0 +1,135 @@
+#!/usr/bin/env bats
+# Pins scripts/check-generate-targets.sh against a rendered seed: the pristine seed
+# must fail for every language (no generator ships in the seed), stubbed generator
+# artifacts must pass, and a target that exits 0 having done nothing must fail.
+# Only `make` is needed: the dry run never executes a generator, and the run-mode
+# cases put a fake `swift` on PATH behind the Swift sub-Makefile's `swift run`.
+
+setup() {
+  tmp="$(mktemp -d)"
+  "$BATS_TEST_DIRNAME/../render-seed.sh" "$tmp" > /dev/null
+  cd "$tmp" || exit 1
+}
+
+teardown() {
+  rm -rf "$tmp"
+}
+
+# The artifacts the rendered Makefile's recipes invoke, as the scaffold step leaves
+# them; package-lock.json because ts-generate-services depends on the install stamp.
+stub_generators() {
+  mkdir -p go/cmd/generate-services typescript/scripts ruby/scripts kotlin/generator rust/generator swift
+  touch go/cmd/generate-services/main.go typescript/scripts/generate-services.ts \
+    typescript/package-lock.json ruby/scripts/generate-services.rb \
+    kotlin/generator/build.gradle.kts rust/generator/Cargo.toml
+  printf 'generate:\n\tswift run FizzyGenerator --output Sources/Fizzy/Generated\n' > swift/Makefile
+}
+
+set_profile() {
+  perl -pi -e "s/^SDK_LANGUAGES := .*/SDK_LANGUAGES := $1/" Makefile
+}
+
+@test "pristine seed: every generate target in the profile is unwired" {
+  run scripts/check-generate-targets.sh --dry-run
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"go/cmd/generate-services missing"* ]]
+  [[ "$output" == *"swift-generate-services does not expand"* ]]
+  [[ "$output" == *"ERROR: generate targets not wired: go ts rb swift kt rs"* ]]
+}
+
+@test "make generate-services-check fails on the pristine seed" {
+  run make generate-services-check
+  [ "$status" -ne 0 ]
+  [[ "$output" == *"ERROR: generate targets not wired: go ts rb swift kt rs"* ]]
+}
+
+@test "stubbed generators pass the dry run, Swift through its sub-Makefile" {
+  stub_generators
+  run scripts/check-generate-targets.sh --dry-run
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"invokes the generator: swift run FizzyGenerator"* ]]
+  [[ "$output" == *"generate targets wired: go ts rb swift kt rs"* ]]
+}
+
+@test "a sub-Makefile generate target with no recipe is vacuous" {
+  stub_generators
+  printf 'generate:\n' > swift/Makefile
+  run scripts/check-generate-targets.sh --dry-run
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"swift-generate-services never invokes the generator"* ]]
+  [[ "$output" == *"ERROR: generate targets not wired: swift"* ]]
+}
+
+@test "a root alias that lost its prerequisite is vacuous" {
+  stub_generators
+  perl -pi -e 's/^swift-generate-services: swift-generate$/swift-generate-services:/' Makefile
+  run scripts/check-generate-targets.sh --dry-run swift
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"swift-generate-services never invokes the generator"* ]]
+}
+
+@test "a recipe that was removed is vacuous even when its prerequisites plan work" {
+  stub_generators
+  # ts-generate-services keeps ts-install, which still plans `npm ci` and the stamp.
+  perl -0pi -e 's/^ts-generate-services:\n\t\@echo[^\n]*\n\tcd typescript && npx tsx scripts\/generate-services\.ts\n/ts-generate-services:\n/m' Makefile
+  ! grep -q 'npx tsx scripts/generate-services.ts' Makefile
+  run scripts/check-generate-targets.sh --dry-run ts
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"npm ci"* ]]
+  [[ "$output" == *"ts-generate-services never invokes the generator"* ]]
+}
+
+@test "a generator named only inside an echo does not count" {
+  stub_generators
+  printf 'generate:\n\t@echo "swift run FizzyGenerator"\n' > swift/Makefile
+  run scripts/check-generate-targets.sh --dry-run swift
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"swift-generate-services never invokes the generator"* ]]
+}
+
+@test "a commented-out generator command does not count" {
+  stub_generators
+  perl -pi -e 's/^\tcd ruby && ruby scripts\/generate-services\.rb$/\t# cd ruby && ruby scripts\/generate-services.rb/' Makefile
+  grep -q '^	# cd ruby && ruby scripts/generate-services.rb' Makefile
+  run scripts/check-generate-targets.sh --dry-run rb
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"rb-generate-services never invokes the generator"* ]]
+}
+
+@test "a narrowed profile is checked for its own languages only" {
+  set_profile "go rs"
+  run scripts/check-generate-targets.sh --dry-run
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"ERROR: generate targets not wired: go rs"* ]]
+  [[ "$output" != *"ts-generate-services"* ]]
+}
+
+@test "an unknown prefix in SDK_LANGUAGES fails make at parse time" {
+  set_profile "go tsx"
+  run make -n conformance
+  [ "$status" -ne 0 ]
+  [[ "$output" == *"SDK_LANGUAGES has unknown prefix(es): tsx"* ]]
+}
+
+@test "run mode executes the target and reports its exit status" {
+  stub_generators
+  mkdir bin
+  printf '#!/bin/sh\ntouch generated.marker\n' > bin/swift
+  chmod +x bin/swift
+  PATH="$PWD/bin:$PATH"
+  run scripts/check-generate-targets.sh swift
+  [ "$status" -eq 0 ]
+  [ -f swift/generated.marker ]
+  [[ "$output" == *"make swift-generate-services exited 0"* ]]
+
+  printf '#!/bin/sh\nexit 3\n' > bin/swift
+  run scripts/check-generate-targets.sh swift
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"make swift-generate-services exited non-zero"* ]]
+}
+
+@test "an unknown prefix is rejected" {
+  run scripts/check-generate-targets.sh --dry-run py
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"unknown language prefix 'py'"* ]]
+}

prompts/seed-sdk.md

@@ -21,6 +21,12 @@ Decide which languages to include based on the target audience:
 
 The profile determines which rubric criteria apply (91 for full-sdk, 76 for single-language). Choose the minimal set that covers your deployment surface, then add languages later if needed.
 
+The Makefile records the choice as `SDK_LANGUAGES`, a list of target prefixes
+(`go ts rb swift kt rs` is the full set), and `make check`, `make generate-services`,
+`make conformance` and `make clean` range over it; per-language targets for languages
+outside the list still exist and still fail. Set it in Phase 3, and keep the
+`conformance` matrix in `.github/workflows/test.yml` to the same languages in Phase 5.
+
 ## Template Variable Reference
 
 All `.tmpl` files use Go template syntax. Replace these placeholders before renaming:
@@ -70,52 +76,60 @@ The spec drives everything downstream.
 
 ### Phase 3: Build pipeline
 
-1. `Makefile.tmpl` -> `Makefile`
+1. `Makefile.tmpl` -> `Makefile`, then set `SDK_LANGUAGES` at the top to the prefixes of
+   the languages chosen above
 2. `scripts/` -- copy build helper scripts
 
 **Checkpoint:** `make smithy-build` produces `openapi.json`.
 
 ### Phase 4: Per-language SDKs
 
-Initialize each language in parallel -- they are independent of each other.
+Initialize each language in parallel -- they are independent of each other. The seed
+ships no generator for any language: each `<prefix>-generate-services` recipe names an
+artifact the scaffold step has to produce, and `scripts/check-generate-targets.sh
+<prefix>` runs the target only after asserting that artifact exists and the target's
+plan invokes it, so a scaffold that quietly produced nothing fails there rather than
+at the first spec change.
 
 #### Go
 1. Copy `seed/go/` into `go/`
 2. `cd go && go mod init {{.ModulePath}}`
-3. Scaffold client, error types, service base
-4. `make go-generate-services`
+3. Scaffold client, error types, service base, and the generator at `go/cmd/generate-services/`
+4. `scripts/check-generate-targets.sh go` (runs `make go-generate-services`)
 
 **Checkpoint:** `make go-check` passes (format + vet + test).
 
 #### TypeScript
 1. Copy `seed/typescript/` into `typescript/`
 2. `cd typescript && npm init --scope={{.NpmScope}}`
-3. Scaffold client, error types, service base
-4. `make ts-generate-services`
+3. Scaffold client, error types, service base, and the generator at `typescript/scripts/generate-services.ts`
+4. `scripts/check-generate-targets.sh ts` (runs `make ts-generate-services`)
 
 **Checkpoint:** `make ts-check` passes (tsc + lint + test).
 
 #### Ruby
 1. Copy `seed/ruby/` into `ruby/`
 2. `cd ruby && bundle init`
-3. Scaffold client, error types, service base
-4. `make rb-generate-services`
+3. Scaffold client, error types, service base, and the generator at `ruby/scripts/generate-services.rb`
+4. `scripts/check-generate-targets.sh rb` (runs `make rb-generate-services`)
 
 **Checkpoint:** `make rb-check` passes (rubocop + test).
 
 #### Swift
 1. Copy `seed/swift/` into `swift/`
 2. Initialize `Package.swift`
-3. Scaffold client, error types, service base
-4. `make swift-generate-services`
+3. Scaffold client, error types, service base, and `swift/Makefile` with a `generate`
+   target that runs the generator with `swift run` -- the root `swift-generate-services`
+   delegates to it
+4. `scripts/check-generate-targets.sh swift` (runs `make swift-generate-services`)
 
 **Checkpoint:** `swift build && swift test` pass.
 
 #### Kotlin
 1. Copy `seed/kotlin/` into `kotlin/`
 2. Initialize `build.gradle.kts`
-3. Scaffold client, error types, service base
-4. `make kt-generate-services`
+3. Scaffold client, error types, service base, and the generator at `kotlin/generator/`
+4. `scripts/check-generate-targets.sh kt` (runs `make kt-generate-services`)
 
 **Checkpoint:** `./gradlew build` passes.
 
@@ -129,7 +143,7 @@ Initialize each language in parallel -- they are independent of each other.
    passes `--locked`, so the lockfile must already know the generator: one generated
    before the workspace gained that member fails the next step with "the lock file
    needs to be updated but --locked was passed".
-4. `make rs-generate-services`
+4. `scripts/check-generate-targets.sh rs` (runs `make rs-generate-services`)
 
 **Checkpoint:** `make rs-check` passes (fmt + clippy + tests + docs + deny + drift +
 publish dry-run). Before the generator exists, `cd rust && cargo test && cargo publish
@@ -139,15 +153,16 @@ publish dry-run). Before the generator exists, `cd rust && cargo test && cargo p
 
 1. Copy `conformance/` from sdk/common (tests + schema)
 2. Initialize per-language conformance runners
-3. Copy `.github/` workflow templates
+3. Copy `.github/` workflow templates; trim the `conformance` matrix in `test.yml` to
+   the languages in `SDK_LANGUAGES`
 4. `rubric-audit.json` -- create initial audit with profile and date
 
 **Checkpoint:** `make conformance` runs (tests may fail -- that's the gap to close).
 
 ### Phase 6: Full verification
 
 ```bash
-make check       # smithy-check + all lang checks + conformance + audit-check
+make check       # smithy-check + generate-services-check + the profile's lang checks + conformance + audit-check
 ```
 
 All checks must pass before the first commit to main.
@@ -168,8 +183,9 @@ Run these per-language to confirm the SDK is functional end-to-end:
 Then cross-language:
 
 ```bash
-make conformance        # All conformance tests
-make audit-check        # rubric-audit.json exists, fresh, must-pass criteria met
+scripts/check-generate-targets.sh   # Every generate target in the profile runs, against a real generator
+make conformance                    # Conformance tests for the profile's languages
+make audit-check                    # rubric-audit.json exists, fresh, must-pass criteria met
 ```
 
 Finally, run the `rubric-audit` skill to establish a baseline score and identify gaps to close.