test(guards): fail when a documented CLI flag is missing from the parser - #370
Open
caasitrube1980 wants to merge 1 commit into
Open
caasitrube1980 wants to merge 1 commit into
caasitrube1980 wants to merge 1 commit into
Conversation
guards/actions-pinned.mjs checks GitHub Actions SHA pins. guards/markdown-links.mjs checks that Markdown links resolve to real files. Neither compares a documentation claim against the CLI's actual behavior, so nothing in this repository would catch a CLI flag documented in apps/docs/docs/reference/cli.md that packages/cli/src/cli.mjs does not accept. This guard reads the `| Flag | Purpose |` table under each `## `facility <command>`` heading in cli.md and the per-command flag allowlist inside validateLocalFlags in cli.mjs, and fails when a documented flag is absent from the real allowlist. It passes against the current repository: cli.md and cli.mjs are consistent today. Verified the guard actually catches drift by adding a fictitious --auth=<mode> row to the cli.md table, confirming the guard fails, then reverting that row. Scope is deliberately narrow: only structured table cells in cli.md are parsed, never README.md or free-flowing prose. Facility's own flag values embed other tools' flags as literal strings (--provision='pnpm install --frozen-lockfile', --preview-readiness-command='curl --fail ...'), so a generic --\w+ scan over prose would misreport those as undocumented Facility flags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adrian-lorenzo
requested changes
Sep 14, 2026
adrian-lorenzo
left a comment
Member
There was a problem hiding this comment.
Thanks for adding this guard—it helps catch documentation drift automatically.
Two cases need tightening: flag tables under unrelated headings are treated as Facility flags, and commented-out allowlist entries still count as accepted. Please restrict parsing to the intended tables, exclude commented-out entries, and add regression tests for both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Adds
guards/documented-cli-flags.mjs, which fails when a CLI flag documentedin the
| Flag | Purpose |table ofapps/docs/docs/reference/cli.mdismissing from the real per-command flag allowlist enforced by
validateLocalFlagsinpackages/cli/src/cli.mjs. Addsscripts/documented-cli-flags-guard.test.mjs(5 unit tests) for the guard'stwo exported parsing helpers, following the existing
scripts/markdown-links-guard.test.mjspattern.After this merges,
node guards/run.mjs(and CI's guard step) also failswhen someone documents a
facility initflag incli.mdthat the real flagparser does not accept.
Why
The repository has two guards today:
actions-pinnedchecks GitHub ActionsSHA pins, and
markdown-linkschecks that Markdown links resolve to realfiles (confirmed by reading both files in full). Neither compares a
documentation claim against the CLI's actual behavior — this adds that class
of check for the first time.
guards/run.mjsstates its own design goal: "adding a new safety invariantis a single small file — easy for engineers and for agents to extend"
(
guards/run.mjs:27-28). This PR is exactly that: one new guard file plusits test.
This is not a fix for a broken state.
cli.mdandcli.mjsare consistentright now:
It's a preventive check for a comparison this repository has never
automated. No linked issue — this came out of reviewing the project rather
than from the tracker.
Scope is deliberately narrow: only the
| Flag | Purpose |table cellsunder a
##facility `` heading incli.mdare parsed — neverREADME.md or general prose. README.md's own CLI examples embed other tools'
flags as literal string values (`--provision='pnpm install
--frozen-lockfile'`, `--preview-readiness-command='curl --fail
http://localhost:3000/health'`); a generic `--\w+` regex over prose would
misreport `--frozen-lockfile` and `--fail` as undocumented Facility flags.
The structured table has no such embedded values in its cells, so this scope
avoids that entire class of false positive. The real allowlist is read as
literal source text out of `cli.mjs` (`command: new Set([...])`), not
imported or executed, keeping the guard read-only and consistent with the
"deterministic, read-only" guard contract in `guards/_kit.mjs`.
Verification
Fault injection, to confirm the guard actually catches what it claims to
catch (applied manually, confirmed, then reverted — not part of this diff):
added
| `--auth=<mode>` | Set the authentication mode. |as afictitious row to the
facility inittable incli.md.New test file:
pnpm typecheck: 14/14 packages pass.pnpm lintandnode --test scripts/*.test.mjshave pre-existing failureson this Windows machine unrelated to this change — confirmed by removing the
two new files and reproducing the same failures.
pnpm lint's 275 errorsare CRLF line-ending diffs on files this PR does not touch (Windows checkout
without
core.autocrlfenforcement); the two new files are not among them.The
node --testfailures (npm-publish/registry, Docker image promotion,gRPC binary audit suites) fail with
tar (child): Cannot connect to C: resolve failed— GNU tar (via Git Bash) misreading a WindowsC:\...pathas a
host:pathremote-shell target — unrelated to this diff.Known risks / follow-up: the guard only checks the
facility inittabletoday, since it's the only command with a
| Flag | Purpose |table incli.md; it extends automatically if a similar table is added fordoctoror
instance bootstraplater. It does not flag the reverse case: a realflag accepted but undocumented.
--orgis ininit's allowlist(
packages/cli/src/cli.mjs:118) but not incli.md's table or infacility --help's flag summary. Catching that direction would need adifferent heuristic to avoid false positives on genuinely
internal/undocumented flags, so it's left out of scope here.
pnpm verifypasses locally — not run to completion. It failsimmediately on this machine with
spawnSync docker ENOENT: Docker isnot installed here, and
pnpm verifystarts by checking/starting aDocker-backed PostgreSQL before anything else runs.
pnpm typecheck,pnpm lint,node guards/run.mjs, andnode --test scripts/*.test.mjswere run individually instead (see above); relying on CI, which has
Docker, for the full
pnpm verifypass.fictitious documented flag (
--auth=<mode>) intocli.mdandconfirmed the guard fails with the expected message and file:line,
then reverted it and confirmed the guard passes again. This exercises
the actual failure path, not just the unit tests of its helpers.
change. This is repository-internal tooling (a guard under
guards/),not a documented product feature; nothing in
apps/docsdescribesguard internals.