Repository navigation
F6c: run every T9 suite in CI and add the live probe workflow - #177
Merged
Merged
Conversation
Remove the Agent isolation workflow's path filter and run it on pushes to the same branches as CI, so `real-docker` can be a required check: a required check that a path filter skips never reports. Stop excluding test/agent-adapter.test.ts from the CI `test` job. It has never started Docker or a vendor CLI: since D4 (#47) its cases test pure parsers and budgets or refuse before anything is allocated, and it passes with no Docker binary on PATH. It was never on vitest.config.ts's Docker list. The excludes now equal that list, and a test fails if they differ or if real-docker gains a path filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…probes The workflow runs on workflow_dispatch and a nightly schedule, never on a pull request or push, so it is not a required check. It reads the repository secret CLAUDE_CODE_OAUTH_TOKEN and fails at once when the secret is empty. It runs only the four live Claude probes, selected by a shared "(live Claude)" title suffix, through the docker Vitest project so the files run one at a time. Vitest exits 0 when it skips every test, so scripts/live-probes.ts then checks the JSON report: every listed probe must pass and no other test may run. Tests pin the triggers, the command, the secret guard, the probe list against the test files, and the report check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add docs/implementation/ci.md: which workflow and job runs which tests on which triggers, the required checks and secret a repository administrator must set, the live probe job, and how to record the planning fixtures again. Link it from the README, agent isolation and planning acceptance, and update the T9 status in the plan. T9 stays unticked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-time note The probe-name check now finds every test inside a CODEBOOST_RUN_AUTH_PROBES gate and requires the "(live Claude)" suffix and a LIVE_PROBES entry, so an unsuffixed gated test cannot be silently skipped by the workflow's -t filter. Unrecognized gate forms fail the check. The workflow comment now says only agent-container and agent-supervisor build the image in beforeAll (53 min of limits, under 75). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parse each live probe file with the Vite (oxc) parser instead of counting braces. The probe switch may only gate an if block of tests and call-free declarations, or it.runIf; the gated tests must match LIVE_PROBES by Vitest full name. The CI test job's npm test command may hold only --exclude <file> pairs after --. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Declare vite 8.3.0 as a direct dev dependency, since the test imports parseSync from it. Count every textual occurrence of the probe switch in each probe file and require each to be a recognized gate; treat a template literal without expressions as a mention. Scan every .ts file under test/, not only *.test.ts, for the switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert that the required test and real-docker jobs and the live-probes job carry no if or continue-on-error at job or step level. Count a probe switch gate only at module load or in a describe callback. Scan every JS/TS source extension under test/ for the switch. Document the 60-day schedule disable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…jobs - Throw when a describe around a gated probe has a non-literal title or a modifier such as .each or .for, instead of computing a wrong full name. - Derive the live probe -t filter and the suffix checks from LIVE_PROBE_MARKER. - Pin the test and real-docker jobs: one job per workflow, keys runs-on, timeout-minutes and steps only (no name, no matrix), exact steps. - Give the CI test job a 45 min timeout (longest recent run: 20 min) and check that all three workflows use the same Node version. - Document that renaming either required job breaks the required checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Allow-list each workflow's top-level keys and the live-probes job keys, and pin the live-probes steps exactly. - Accept a probe gate only as a statement directly in the module body or in a describe callback body, not inside another if, block or expression. - Docs: say "variable declarations", list the pinned steps of each job. - live-probes.yml: include the teardown hooks in the timeout arithmetic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…teardown comment - A gated if block may declare only const or let names with a function expression or a constant value; a var or an assignment such as (on = true) now fails the config test. - Pin the live-probes token step's run text exactly. - Say the teardown figures are Vitest's limits, that the afterAll docker calls have no timeout of their own, and that the job limit caps the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d titles in probe files The scheduling test now compares the whole Vitest config, so an extra key such as an exclude fails it. Gated tests and the describes around them must take exactly a literal title, a function and an optional timeout. Every it/test/describe title in a probe file must be a literal, so the marker check sees each one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…deled links
A call such as it.each`table`('title', fn) has a tagged template in its callee
chain, so the title check skipped it. suiteCall now throws when the chain reaches
it, test or describe through a tagged template, computed member, parentheses or
a TypeScript wrapper. it.each([...])(title, fn) stays modeled with the literal
title check. The comment and ci.md now state exactly what is enforced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set forbidOnly: true in playwright.config.ts and pin the whole config in test/vitest-config.test.ts, so a committed .only, a grep or a narrower testDir fails the test job. In ci.md, state that only top-level test/*.test.ts files run, and that the static probe-title check does not see a marker added by a .each/.for placeholder (the live-probes report check refuses that test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Exact-head required CI results remain pending, and the new credentialed workflow lacks execution evidence.
No open issues 🎉
What changed in this PR
Completes F6c’s CI wiring for T9 after F6a/F6b, while leaving T9 open pending configuration and passing runs.
Changes:
- Runs both required-check workflows on every pull request.
- Adds nightly/manual Claude probes with explicit result verification.
- Adds configuration drift checks, browser
.onlyprotection, and CI documentation.
| File | Description |
|---|---|
test/vitest-config.test.ts |
Guards workflow, configuration, and probe coverage. |
test/agent-supervisor.test.ts |
Marks two live Claude probes. |
test/agent-container.test.ts |
Marks the authenticated startup probe. |
scripts/live-probes.ts |
Lists probes and verifies their results. |
README.md |
Links CI documentation. |
playwright.config.ts |
Rejects focused-only browser tests. |
package.json |
Declares Vite for source parsing. |
package-lock.json |
Updates direct dependency metadata. |
docs/implementation/planning-acceptance.md |
Clarifies remaining T9 gates. |
docs/implementation/ci.md |
Documents workflows and administrator setup. |
docs/implementation/agent-isolation.md |
Updates CI and live-probe guidance. |
docs/designs/codeboost-plan-indexed-review.md |
Records F6c progress without completing T9. |
.github/workflows/live-probes.yml |
Adds credentialed nightly/manual probes. |
.github/workflows/ci.yml |
Includes adapter tests and bounds job duration. |
.github/workflows/agent-isolation.yml |
Removes path filters and aligns push triggers. |
Review effort: Balanced
mchwang
marked this pull request as ready for review
October 18, 2026 09:25
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #22 (F6c, common CI / T9). F6a (#153) and F6b (#161) are merged.
Summary
Every T9 suite now runs on every pull request, so
testandreal-dockercan be required checks. The live Claude startup probes run in their own nightly and manual workflow. Decisions (made with the user, 2026-10-18): keep both jobs; dropreal-docker's path filter; run the live probes nightly and by hand with a repository secret, not as a required check; a manual re-record of the Claude planning fixtures is enough; Codex stays refused in every phase (#93).ci.yml/test: type check, every top-level Vitest file except the seven Docker-backed ones, then the Playwright browser suite.agent-adapter.test.tsis no longer excluded: it has needed neither Docker nor a vendor CLI since D4: Add bounded production agent adapters #47.agent-isolation.yml/real-docker: no path filter any more; same triggers astest(every pull request; pushes tomainandcodex/**). Runs the full Vitest suite, Docker-backed files one at a time.live-probes.yml/live-probes(new):workflow_dispatchand nightly at 06:23 UTC; never on a pull request or push. Fails at once whenCLAUDE_CODE_OAUTH_TOKENis empty, runs only the tests titled "(live Claude)", andscripts/live-probes.ts checkfails unless every listed probe passed and nothing else ran (Vitest exits 0 when every test is skipped).playwright.config.ts:forbidOnly: true, so a committedtest.onlycannot narrow the browser suite.test/vitest-config.test.tskeeps this from drifting: the Docker excludes equal the Docker list; the Vitest and Playwright configs, the three npm scripts, each workflow's top-level keys, triggers, job keys and steps are pinned; no gating job or step may be skipped or soft-fail; the job ids (the required check names) cannot change; and every test the probe switch enables must be a listed probe at a place Vitest registers it. Its documented limits: a switch name built at runtime, a call through an alias, and.eachtitles before placeholder formatting (the nightly report check catches the last).vite8.3.0 becomes a direct dev dependency (it was already installed as Vitest's peer at that version), because the config test parses the probe files with it.docs/implementation/ci.md; T9 status updated in the design plan,agent-isolation.mdandplanning-acceptance.md. T9 is not ticked.Your actions (I do not change repository settings)
main, require the checkstestandreal-docker. Do not requirelive-probes.CLAUDE_CODE_OAUTH_TOKEN(fromclaude setup-token).live-probesrun by hand (schedules run only frommain).T9 can be ticked once both required checks pass on the combined head and a
live-probesrun is green.Validation (head
bdd9bac, basemain04b9973)npm run typechecktest/vitest-config.test.ts+test/agent-adapter.test.tsdockeronPATH)PATH, at88dcbf5)first-load.spec.tswith the new Playwright configtest,real-dockeron this headtest19m46s;real-docker22m59s, its first run on a pull request without a path filter)live-probesmainReview record
88dcbf5acb744aacb744a9e5b9599e5b959viteundeclared; template-literal switch name; helpers outside*.test.ts465b007465b007ifandcontinue-on-errorunpinned; dead-code gates counted; scan scope; public-repo schedule notebe37122be37122teststeps unpinned;testtimeoute080b5ee080b5edefaults; nested gates; doc wording; timeout arithmetic2225d0c2225d0cvar/assignment in a gated block; token step text; teardown comment125f5fb125f5fbfd85031fd85031.eachcalls unseen9a69e879a69e87package.jsontest scripts unpinnedf615b25f615b25test.onlyin browser specs; Playwright config unpinned;.eachwording; top-level-only includebdd9bacbdd9bacbdd9baclive-probeshas no run yet (needs the secret and the merge, as above).Declined: none. Every test fix was shown failing under the mutation it guards against.
Review-lesson audit
🤖 Generated with Claude Code