Skip to content

test(guards): fail when a documented CLI flag is missing from the parser - #370

Open
caasitrube1980 wants to merge 1 commit into
theam:mainfrom
caasitrube1980:feature/documented-cli-flags-guard
Open

caasitrube1980 wants to merge 1 commit into
theam:mainfrom
caasitrube1980:feature/documented-cli-flags-guard

Conversation

@caasitrube1980

Copy link
Copy Markdown

What changes

Adds guards/documented-cli-flags.mjs, which fails when a CLI flag documented
in the | Flag | Purpose | table of apps/docs/docs/reference/cli.md is
missing from the real per-command flag allowlist enforced by
validateLocalFlags in packages/cli/src/cli.mjs. Adds
scripts/documented-cli-flags-guard.test.mjs (5 unit tests) for the guard's
two exported parsing helpers, following the existing
scripts/markdown-links-guard.test.mjs pattern.

After this merges, node guards/run.mjs (and CI's guard step) also fails
when someone documents a facility init flag in cli.md that the real flag
parser does not accept.

Why

The repository has two guards today: actions-pinned checks GitHub Actions
SHA pins, and markdown-links checks that Markdown links resolve to real
files (confirmed by reading both files in full). Neither compares a
documentation claim against the CLI's actual behavior — this adds that class
of check for the first time.

guards/run.mjs states its own design goal: "adding a new safety invariant
is a single small file — easy for engineers and for agents to extend"
(guards/run.mjs:27-28). This PR is exactly that: one new guard file plus
its test.

This is not a fix for a broken state. cli.md and cli.mjs are consistent
right now:

$ node guards/run.mjs
✓ actions-pinned
✓ documented-cli-flags
✓ markdown-links

3 guards ran, 0 failed.

It's a preventive check for a comparison this repository has never
automated. No linked issue — this came out of reviewing the project rather
than from the tracker.

Scope is deliberately narrow: only the | Flag | Purpose | table cells
under a ## facility `` heading in cli.md are parsed — never
README.md or general prose. README.md's own CLI examples embed other tools'
flags as literal string values (`--provision='pnpm install
--frozen-lockfile'`, `--preview-readiness-command='curl --fail
http://localhost:3000/health'`); a generic `--\w+` regex over prose would
misreport `--frozen-lockfile` and `--fail` as undocumented Facility flags.
The structured table has no such embedded values in its cells, so this scope
avoids that entire class of false positive. The real allowlist is read as
literal source text out of `cli.mjs` (`command: new Set([...])`), not
imported or executed, keeping the guard read-only and consistent with the
"deterministic, read-only" guard contract in `guards/_kit.mjs`.

Verification

Fault injection, to confirm the guard actually catches what it claims to
catch (applied manually, confirmed, then reverted — not part of this diff):
added | `--auth=<mode>` | Set the authentication mode. | as a
fictitious row to the facility init table in cli.md.

$ node guards/run.mjs
✓ actions-pinned
✗ documented-cli-flags
    apps/docs/docs/reference/cli.md:36  --auth is documented for `facility init` but is not in the real flag allowlist (packages/cli/src/cli.mjs)
✓ markdown-links

3 guards ran, 1 failed.

New test file:

$ node --test scripts/documented-cli-flags-guard.test.mjs
✔ documentedFlags reads only table cells under a `facility <command>` heading
✔ documentedFlags records the doc line number for each flag
✔ realFlags reads the quoted flag names out of a command's allowlist Set
✔ realFlags returns null for a command absent from the allowlist
✔ a flag documented but missing from the allowlist is exactly what the guard must catch
ℹ tests 5, pass 5, fail 0

pnpm typecheck: 14/14 packages pass.

pnpm lint and node --test scripts/*.test.mjs have pre-existing failures
on this Windows machine unrelated to this change — confirmed by removing the
two new files and reproducing the same failures. pnpm lint's 275 errors
are CRLF line-ending diffs on files this PR does not touch (Windows checkout
without core.autocrlf enforcement); the two new files are not among them.
The node --test failures (npm-publish/registry, Docker image promotion,
gRPC binary audit suites) fail with tar (child): Cannot connect to C: resolve failed — GNU tar (via Git Bash) misreading a Windows C:\... path
as a host:path remote-shell target — unrelated to this diff.

Known risks / follow-up: the guard only checks the facility init table
today, since it's the only command with a | Flag | Purpose | table in
cli.md; it extends automatically if a similar table is added for doctor
or instance bootstrap later. It does not flag the reverse case: a real
flag accepted but undocumented. --org is in init's allowlist
(packages/cli/src/cli.mjs:118) but not in cli.md's table or in
facility --help's flag summary. Catching that direction would need a
different heuristic to avoid false positives on genuinely
internal/undocumented flags, so it's left out of scope here.

  • pnpm verify passes locally — not run to completion. It fails
    immediately on this machine with spawnSync docker ENOENT: Docker is
    not installed here, and pnpm verify starts by checking/starting a
    Docker-backed PostgreSQL before anything else runs. pnpm typecheck,
    pnpm lint, node guards/run.mjs, and node --test scripts/*.test.mjs
    were run individually instead (see above); relying on CI, which has
    Docker, for the full pnpm verify pass.
  • Behaviour verified beyond the test suite (say how) — fault-injected a
    fictitious documented flag (--auth=<mode>) into cli.md and
    confirmed the guard fails with the expected message and file:line,
    then reverted it and confirmed the guard passes again. This exercises
    the actual failure path, not just the unit tests of its helpers.
  • Documentation updated, or no user-facing change — no user-facing
    change. This is repository-internal tooling (a guard under guards/),
    not a documented product feature; nothing in apps/docs describes
    guard internals.

guards/actions-pinned.mjs checks GitHub Actions SHA pins. guards/markdown-links.mjs
checks that Markdown links resolve to real files. Neither compares a documentation
claim against the CLI's actual behavior, so nothing in this repository would catch
a CLI flag documented in apps/docs/docs/reference/cli.md that
packages/cli/src/cli.mjs does not accept.

This guard reads the `| Flag | Purpose |` table under each `## `facility
<command>`` heading in cli.md and the per-command flag allowlist inside
validateLocalFlags in cli.mjs, and fails when a documented flag is absent from
the real allowlist.

It passes against the current repository: cli.md and cli.mjs are consistent
today. Verified the guard actually catches drift by adding a fictitious
--auth=<mode> row to the cli.md table, confirming the guard fails, then
reverting that row.

Scope is deliberately narrow: only structured table cells in cli.md are
parsed, never README.md or free-flowing prose. Facility's own flag values
embed other tools' flags as literal strings (--provision='pnpm install
--frozen-lockfile', --preview-readiness-command='curl --fail ...'), so a
generic --\w+ scan over prose would misreport those as undocumented Facility
flags.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this guard—it helps catch documentation drift automatically.

Two cases need tightening: flag tables under unrelated headings are treated as Facility flags, and commented-out allowlist entries still count as accepted. Please restrict parsing to the intended tables, exclude commented-out entries, and add regression tests for both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants