Skip to content

fix(ci-queue-watch): a blind watchdog must be RED, not green - #797

Merged
izzywdev merged 2 commits into
masterfrom
claude/ci-queue-watch-blind-is-red
Aug 26, 2026
Merged

fix(ci-queue-watch): a blind watchdog must be RED, not green#797
izzywdev merged 2 commits into
masterfrom
claude/ci-queue-watch-blind-is-red

Conversation

@github-actions

@github-actions github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

📋 Description

The fleet CI watchdog warned on a missing FLEET_READ_PAT, set a step output, and gated its observation step on that output. A scheduled run with no token therefore skipped the work and the job went green.

It did that ten times, every one green — including runs at 07:20Z and 07:57Z while the fleet was 40 hours into an outage the watchdog exists to detect. Green read as "the fleet is fine". It actually meant "I looked at nothing".

That is the exact vacuous pass this tool was built to find, reproduced inside the detector.

The original reasoning was half right, and that half is kept

The old comment said: never red a scheduled run on a missing secret — an environmental condition must not look like a fleet outage. That is correct, and it is preserved verbatim.

What was wrong was the conclusion. The fix is not to go green; it is to fail with a different signal:

Condition Signal
No FLEET_READ_PAT ::error title=Watchdog not configured::… This is a CONFIGURATION failure and NOT a fleet outage
Fleet actually stalled ci_queue_watch.py's stall error

If those two were confusable, the red would be as uninformative as the green was. Hence the distinct title and the explicit disclaimer — an operator must not act on the wrong incident.

Why a missing token here is never transient

The watch job's if: excludes pull_request, leaving schedule (always the default branch, full secrets) and workflow_dispatch (requires write access). Neither can legitimately arrive without repo secrets. A missing secret in this job is misconfiguration, full stop.

The script is deliberately left alone

scripts/ci_queue_watch.py still exits 0 with no token, and test_no_token_skips_cleanly still pins it. That is correct for a CLI someone runs by hand — with no credential there is nothing to misconfigure. The difference is run context, which only the workflow knows. The comment says not to "align" them.

🔄 Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)

🧪 Testing

  • Unit tests

Three new tests in scripts/__tests__/test_ci_queue_watch.py, asserted against the real workflow file rather than a fixture — a fixture would have to encode the structure under test and could therefore agree with the bug, which is the failure mode that let a hardcoded default_branch survive twelve green tests elsewhere in this repo.

Mutation-verified. Each mutation is caught by the assertion written for it:

revert guard to warn-and-flag   -> FAILED (failures=2)
restore `if: steps.tok...`      -> FAILED (failures=1)
drop the outage disclaimer      -> FAILED (failures=1)
restored                        -> Ran 18 tests, OK; workflow byte-identical

One process note worth recording

My first attempt at adding these tests silently deleted them. A dedupe of the __main__ block cut out the class I had just inserted; the suite reported 18 test methods as 15 and went green. Caught by counting, not by the suite.

The lesson generalises, and it is the same one this PR is about: a green suite is evidence only if you know what it ran.

🔧 Implementation Details

Changes Made

  • CI: ci-queue-watch.yml — the token step now exit 1s with a configuration-specific error instead of exporting a skip flag; the observation step's if: gate is removed so it cannot be silently bypassed.
  • Tests: three assertions covering the guard's failure mode, the absence of any token-presence gate, and the distinguishability of the two red signals.

No application code, no script behaviour change.

🚨 Breaking Changes

The watchdog will go red on its next scheduled run until FLEET_READ_PAT is set. That is the intended behaviour and the entire point of the change — it is currently reporting green while blind. The error explicitly states it is a configuration failure and not a fleet outage.

If that 15-minute red is unwanted before the secret lands, the honest lever is disabling the schedule — not restoring the green.

📋 Checklist

  • I have performed a self-review of my code
  • I have added tests that prove my fix is effective
  • New and existing unit tests pass locally with my changes

🔗 Related Issues and PRs

📝 Additional Notes

Deployment Notes

This repo is hardening.deployOnPush: true, and the auto-merge label was applied automatically on push. Per #771 that is now the intended path here rather than something to fight — so this PR will merge itself once the required checks pass, and dispatch-release will ship it.

Flagging it explicitly because of the consequence above: merging starts a red watchdog every 15 minutes until FLEET_READ_PAT exists. If you would rather set the secret first, remove the auto-merge label before CI goes green.

Future Work

Setting FLEET_READ_PAT is the actual remediation; this PR only ensures its absence is visible rather than silently green.

🔄 Backwards Compatibility

  • Fully backwards compatible (no interface change; only the failure signal changes)

The watchdog warned on a missing FLEET_READ_PAT, set an output, and gated its
observation step on that output. So a scheduled run with no token skipped the
work and the JOB WENT GREEN.

It did that ten times. Every one green. Two of them — 07:20Z and 07:57Z — while
the fleet was 40 hours into an outage the watchdog exists to detect. Green read
as "the fleet is fine"; it actually meant "I looked at nothing". That is the
exact vacuous pass this tool was built to find, reproduced inside the detector.

The original reasoning was half right, and that half is preserved verbatim in
the comment: an environmental gap must not masquerade as a FLEET OUTAGE. What
was wrong was the conclusion. The fix is not to go green, it is to fail with a
DIFFERENT SIGNAL — this error says the watchdog is blind, ci_queue_watch.py's
stall error says the fleet is stuck, and they must never be confusable or the
red is as uninformative as the green was.

There is no legitimate no-token run of this job: the job-level `if` excludes
pull_request, leaving schedule (default branch, full secrets) and
workflow_dispatch (requires write access). A missing secret here is
misconfiguration, never a transient.

scripts/ci_queue_watch.py still exits 0 with no token and
test_no_token_skips_cleanly still pins that — correct for a CLI run by hand,
where there is no credential to misconfigure. The difference is RUN CONTEXT,
which only the workflow knows. The comment says not to "align" them.

Three tests, asserted against the REAL workflow file rather than a fixture,
because a fixture would have to encode the structure under test and could agree
with the bug — the failure mode that let a hardcoded default_branch survive
twelve green tests elsewhere in this repo. Mutation-verified, each mutation
caught by the assertion written for it:

  revert guard to warn-and-flag  -> 2 failures
  restore `if: steps.tok...`     -> 1 failure
  drop the outage disclaimer     -> 1 failure
  restored                       -> 18/18 OK, workflow byte-identical

Worth recording: my first attempt at adding these tests silently deleted them —
a dedupe of the `__main__` block cut out the class I had just inserted, and the
suite reported 18 tests as 15 and went green. Caught by counting, not by the
suite. The lesson generalises: a green suite is evidence only if you know what
it ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GaPa3JgrVNtWrGvqQEAEqv
@github-actions
github-actions Bot requested a review from izzywdev as a code owner August 24, 2026 08:32
@github-actions github-actions Bot added the auto-merge Enable squash auto-merge once CI passes label Aug 24, 2026

@izzywdev izzywdev left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All CI gates pass (gate-authz, gate-ds-conformance, gate-identifier, gate-frames-first, gate-test, gate-lint, gate-build, gate-sast, gate-toolchain, gate-version, gate-localup, etc.). Approving per governance policy.

@github-actions
github-actions Bot enabled auto-merge (squash) August 26, 2026 05:01
@izzywdev
izzywdev merged commit 057cb55 into master Aug 26, 2026
56 checks passed
@izzywdev
izzywdev deleted the claude/ci-queue-watch-blind-is-red branch August 26, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge Enable squash auto-merge once CI passes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants