fix(ci-queue-watch): a blind watchdog must be RED, not green - #797
Merged
Conversation
The watchdog warned on a missing FLEET_READ_PAT, set an output, and gated its observation step on that output. So a scheduled run with no token skipped the work and the JOB WENT GREEN. It did that ten times. Every one green. Two of them — 07:20Z and 07:57Z — while the fleet was 40 hours into an outage the watchdog exists to detect. Green read as "the fleet is fine"; it actually meant "I looked at nothing". That is the exact vacuous pass this tool was built to find, reproduced inside the detector. The original reasoning was half right, and that half is preserved verbatim in the comment: an environmental gap must not masquerade as a FLEET OUTAGE. What was wrong was the conclusion. The fix is not to go green, it is to fail with a DIFFERENT SIGNAL — this error says the watchdog is blind, ci_queue_watch.py's stall error says the fleet is stuck, and they must never be confusable or the red is as uninformative as the green was. There is no legitimate no-token run of this job: the job-level `if` excludes pull_request, leaving schedule (default branch, full secrets) and workflow_dispatch (requires write access). A missing secret here is misconfiguration, never a transient. scripts/ci_queue_watch.py still exits 0 with no token and test_no_token_skips_cleanly still pins that — correct for a CLI run by hand, where there is no credential to misconfigure. The difference is RUN CONTEXT, which only the workflow knows. The comment says not to "align" them. Three tests, asserted against the REAL workflow file rather than a fixture, because a fixture would have to encode the structure under test and could agree with the bug — the failure mode that let a hardcoded default_branch survive twelve green tests elsewhere in this repo. Mutation-verified, each mutation caught by the assertion written for it: revert guard to warn-and-flag -> 2 failures restore `if: steps.tok...` -> 1 failure drop the outage disclaimer -> 1 failure restored -> 18/18 OK, workflow byte-identical Worth recording: my first attempt at adding these tests silently deleted them — a dedupe of the `__main__` block cut out the class I had just inserted, and the suite reported 18 tests as 15 and went green. Caught by counting, not by the suite. The lesson generalises: a green suite is evidence only if you know what it ran. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv
izzywdev
approved these changes
Aug 26, 2026
izzywdev
left a comment
Owner
There was a problem hiding this comment.
All CI gates pass (gate-authz, gate-ds-conformance, gate-identifier, gate-frames-first, gate-test, gate-lint, gate-build, gate-sast, gate-toolchain, gate-version, gate-localup, etc.). Approving per governance policy.
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📋 Description
The fleet CI watchdog warned on a missing
FLEET_READ_PAT, set a step output, and gated its observation step on that output. A scheduled run with no token therefore skipped the work and the job went green.It did that ten times, every one green — including runs at 07:20Z and 07:57Z while the fleet was 40 hours into an outage the watchdog exists to detect. Green read as "the fleet is fine". It actually meant "I looked at nothing".
That is the exact vacuous pass this tool was built to find, reproduced inside the detector.
The original reasoning was half right, and that half is kept
The old comment said: never red a scheduled run on a missing secret — an environmental condition must not look like a fleet outage. That is correct, and it is preserved verbatim.
What was wrong was the conclusion. The fix is not to go green; it is to fail with a different signal:
FLEET_READ_PAT::error title=Watchdog not configured::… This is a CONFIGURATION failure and NOT a fleet outageci_queue_watch.py's stall errorIf those two were confusable, the red would be as uninformative as the green was. Hence the distinct title and the explicit disclaimer — an operator must not act on the wrong incident.
Why a missing token here is never transient
The
watchjob'sif:excludespull_request, leavingschedule(always the default branch, full secrets) andworkflow_dispatch(requires write access). Neither can legitimately arrive without repo secrets. A missing secret in this job is misconfiguration, full stop.The script is deliberately left alone
scripts/ci_queue_watch.pystill exits 0 with no token, andtest_no_token_skips_cleanlystill pins it. That is correct for a CLI someone runs by hand — with no credential there is nothing to misconfigure. The difference is run context, which only the workflow knows. The comment says not to "align" them.🔄 Type of Change
🧪 Testing
Three new tests in
scripts/__tests__/test_ci_queue_watch.py, asserted against the real workflow file rather than a fixture — a fixture would have to encode the structure under test and could therefore agree with the bug, which is the failure mode that let a hardcodeddefault_branchsurvive twelve green tests elsewhere in this repo.Mutation-verified. Each mutation is caught by the assertion written for it:
One process note worth recording
My first attempt at adding these tests silently deleted them. A dedupe of the
__main__block cut out the class I had just inserted; the suite reported 18 test methods as 15 and went green. Caught by counting, not by the suite.The lesson generalises, and it is the same one this PR is about: a green suite is evidence only if you know what it ran.
🔧 Implementation Details
Changes Made
ci-queue-watch.yml— the token step nowexit 1s with a configuration-specific error instead of exporting a skip flag; the observation step'sif:gate is removed so it cannot be silently bypassed.No application code, no script behaviour change.
🚨 Breaking Changes
The watchdog will go red on its next scheduled run until
FLEET_READ_PATis set. That is the intended behaviour and the entire point of the change — it is currently reporting green while blind. The error explicitly states it is a configuration failure and not a fleet outage.If that 15-minute red is unwanted before the secret lands, the honest lever is disabling the schedule — not restoring the green.
📋 Checklist
🔗 Related Issues and PRs
gate-secret-scanred-on-clean fix (ci: two required checks on master cannot deliver a correct verdict — unfilter workspace-deps, de-action gate-secret-scan #794): a check whose verdict is decided by something other than what it checks.CLAUDE.mdso that auto-merge →dispatch-releaseis the documented, intended path on this repo.📝 Additional Notes
Deployment Notes
This repo is
hardening.deployOnPush: true, and theauto-mergelabel was applied automatically on push. Per #771 that is now the intended path here rather than something to fight — so this PR will merge itself once the required checks pass, anddispatch-releasewill ship it.Flagging it explicitly because of the consequence above: merging starts a red watchdog every 15 minutes until
FLEET_READ_PATexists. If you would rather set the secret first, remove theauto-mergelabel before CI goes green.Future Work
Setting
FLEET_READ_PATis the actual remediation; this PR only ensures its absence is visible rather than silently green.🔄 Backwards Compatibility