Register the ten distribution surfaces and test that they agree (#497)

[pengfei-threemoonslab]

What this is

The first increment of #497: a registry of every surface this engine is published through, and a parity test that they agree with the engine.

docs/distribution-surfaces.md lists all ten surfaces from the issue plus the two the first increment names (init --write --ci's emitted workflow, and docs/design-partner-verifier-pilot.md) — what each claims, which test proves it, and where it is deliberately narrower than the CLI. tests/test_distribution_surface_parity.py is the machine half; the two are checked against each other, so neither can be edited alone. CONTRIBUTING.md § Surface discipline and CLAUDE.md both point at it.

The invariant, in both halves:

Every surface that answers a question the engine also answers must give the engine's answer, or say in the registry what it does not answer.

The four surfaces that were disagreeing

These are repairs, not just guards.

1. The bundled setup prompt told a coding agent to write a declaration only a person may make. add-shipgate-to-repo.md step 5 said to replace agent.declared_purpose[] with "a one-line description of what the agent should do", derived from the prompt or main module. The engine's answer for that field:

control.next_action.actor: "human"
control.control_state:     "human_review_required"
control.permissions.edit:  false
control.reason: "…agent.declared_purpose[0] must be supplied by a human. These
  fields declare what this agent is for … a value a coding agent supplied is a
  declaration nobody made."

The prompt now separates the placeholder the agent owns (agent.name) from the one it must surface, and quotes the engine's own wording. The Codex kit's recipes page and the design-partner runbook carried the same thing as a blanket "replace every CHANGE_ME".

2. The Claude Code kit rendered a GitHub Action tag that does not exist. Its advisory CI recipe pinned @v{{ shipgate_version }} and shipgate_version: '{{ shipgate_version }}', so init --agent-instructions=claude-code-skill wrote @v0.16.0 and agents-shipgate==0.16.0 into an adopter's CI. Neither is published. GitHub resolves uses: before any step runs, so the adopter's first Shipgate run fails on our repository's name.

It was invisible because test_claude_code_skill_source_matches_renderer skipped that one file, so the checked-in mirror could go on saying v0.15.0 while the renderer emitted v0.16.0. Both pins now name the published release — as the Codex kit's byte-identical recipe always did — and the exemption is deleted, which is the actual repair.

3. Two published surfaces demanded a contract nobody could reach. plugins/claude-code/.claude-plugin/plugin.json and .claude-plugin/marketplace.json both said "runtime contract 15" beside pipx install agents-shipgate, which yields contract 10. The bundled skill already states the floor it needs, so the duplicate is removed rather than re-synced.

4. The design-partner runbook taught a route its own build could not run. It named v0.15.0, demanded "runtime contract 14" (that build implements 10, and never implemented 14), floored pip at >=0.13, and then gave a read order starting at control.state — which shipgate.agent_handoff/v1 does not emit at all. It now names one channel per partner (released / unqualified preview / source checkout) with the contract each implements, and states what the released build does not produce.

The exemption mechanism was tested for real

#485 and #506 both landed while this was in review. All three registered gaps flipped to XPASS, their strict markers failed, and the exemptions had to be removed to get back to green. KNOWN_GAPS is empty because the mechanism worked; the registry keeps a ### Closed record of what each gap was rather than dropping them silently.

The same merge exposed a duplicate — which is the defect this PR exists to catch, so it is removed rather than reconciled:

This branch had	#506 shipped	Now
PUBLISHED_BUILDS, published_version(), contract_of(), floor_out_of_reach()	agents_shipgate.published_release	Deleted; constants imported
PIN_PATTERNS (incl. shipgate_version input, arrived at independently)	test_adopter_pins_resolve.PIN_SHAPES + READER_BLANK_REFS	Imported
A regex scan for contract-floor prose	contract_floor_prose, rendered and proved against the tag	Deleted outright — scanning the older prose would have been a second implementation that was also wrong
test_published_build_table_matches_the_tag, emitted-workflow checks	#506's equivalents	Deleted; the registry names its tests as the proving tests
tests/fixtures/distribution_parity/ts_registertool_positive	samples/mcp_source_only_server	Deleted

What survives is the half #506 does not reach: pins committed under a registered surface, found by path rather than from a list, so an example or runbook added later is covered without anyone enumerating it. The Go workspace and the no-dependency negative stay in the corpus for the same reason — samples/ still has no Go MCP server, and the source-text corpus does not reach the whole-workspace dependency gate.

Importing #506's stronger ref shape immediately caught examples/github-actions/10-check-run-annotations.yml naming @main. That turns out to be the one case #497's own rule allows — an explicit version incompatibility rather than an unresolvable pin, and the file says so in its header. It is now an enumerated declared exception whose file must state which capability postdates the release and what to do once one carries it, so an unexplained @main elsewhere still fails.

Workflow conflicts went main's way: fetch-tags: true without fetch-depth: 0 is sufficient for the tag reads, so the generalized checkout contract asserts that instead of the deeper fetch this branch asked for. The contract itself is still worth having — ci.yml and release-verify.yml were fixed one at a time twice now, once by each side.

Known gaps are rows, not omissions

A surface allowed to diverge gets a KNOWN_GAPS entry, a row in the registry's Known parity gaps table with an owning issue, and xfail(strict=True). It fails today; the day the owning fix lands the row passes, the strict marker turns that into a failure, and the gap has to be retired. A gap cannot rot unnoticed, which is the property the previous arrangement lacked.

Gap	Owner
detector-mcp-server-source — the detector answers false for MCP servers the CLI accepts	#485
emitted-workflow-unpublished-pin — init --ci emits @v<__version__>	#506
rendered-prompt-unpublished-pin — rendered prompts pin uvx agents-shipgate@<emitting build>	#506

#485's exact case is a row that fails before the fix and passes after. tests/fixtures/distribution_parity/ holds minimized TypeScript and Go MCP servers whose tool surface exists only as registration sites — the shape no samples/ fixture carries, which is why the existing parity test could not see the divergence. The CLI reports both as agent projects with one mcp_server_source; the detector reports false. Shared with #485's conformance corpus.

The third fixture is the load-bearing one: the TypeScript idiom spelled exactly, with no declared MCP dependency. Both implementations answer false, and the port has to keep doing so — a proof resting on a spelling is the fail-open shape #393 named.

The pin gaps use a ledger of exactly the diverging files, not a per-surface exemption. A surface-wide excuse would cover the next file to drift, which is how this became invisible; test_unpublished_pin_ledger_is_exact makes a new one fail loudly and a repaired one fail its own strict marker.

What the review round changed

Reviewing the first commit found seven real defects, and the two most important were in the registry itself:

• The harness row named a proving test that did not exist. It had been renamed during self-review, so the document's whole product — "which test proves this claim" — was already false in the change that introduces it, and nothing noticed. Claims are now keyed to their proving tests in code, test_every_claim_names_a_test_that_exists resolves each name against this module, and test_every_scanned_claim_actually_has_rows catches the weaker version: a claim naming a real test whose scan matches no file on that surface. That immediately caught design_partner_runbook's executable_pin, which carried only a >= install floor that no pin pattern looked at — so floors are now checked too, for reachability rather than equality.
• The doc/code comparison ran one way. The registry could advertise a claim the code did not register, and github_action's roots differed between the two with nothing comparing them. Roots, claims and proving tests are now compared as sets in both directions.

Also fixed: the tag corroboration read CONTRACT_VERSION with an unanchored regex (CONTRACT_VERSION is a suffix of MINIMUM_CONTROL_CONTRACT_VERSION, so it survived only because v0.15.0 has no minimum and line 155 precedes 156); ParityGap held one surface id for a gap spanning four, so the xfail reason pointed readers at the wrong place; the ownership guard only ran on files containing the literal CHANGE_ME; and the file walkers read the working tree while the classifier read git ls-files.

Two guards were replaced rather than repaired. The first draft's verdict scan matched nothing anywhere in the repository — a regex that finds nothing reads exactly like a regex that finds no problem. test_surface_enumerations_match_the_engine_vocabulary replaces it, keyed to braced set literals (a comma-joined run is usually a correct partial statement, and flagging those would force prose to be reworded for a test) with the two overlapping vocabularies disambiguated. The contract-floor table was hand-written and covered two of the four shipped copies; it is now scanned, and spelling-agnostic, so the plain-JSON "runtime contract 15" that shipped in plugin.json would be caught. Both have explicit not-vacuous assertions, as does the ownership guard.

Every guard was replayed against the pre-fix text from main and catches the defect it was written for:

prompts/add-shipgate-to-repo.md          -> ['agent.declared_purpose at offset 7630']
docs/design-partner-verifier-pilot.md    -> ["blanket placeholder instruction: 'Replace every CHANGE_ME'"]
adoption-kits/codex-skill/…/recipes.md   -> ["blanket placeholder instruction: 'replace `CHANGE_ME` values'"]
adoption-kits/…/advisory-pr-comment.yml  -> [('action', '0.16.0')]

The two new registry guards were mutation-tested the same way.

Review findings addressed

Three findings on dbda114d, all reproduced before fixing and replayed after:

• [P1] release-verify.yml still checked out shallow and tagless, then ran the whole suite. Requiring the tag when CI is set landed in ci.yml only, and release.yml/release-rehearsal.yml both call release-verify.yml — so the release path would have gone red on a green PR. That checkout now fetches full history and tags, and test_jobs_running_the_whole_suite_check_out_history_and_tags asserts the contract for every job that runs the suite, in any workflow. It classifies by whether the pytest invocation has a positional path, so --ignore=tests/x is not misread as scoping the run; both suite jobs fail it when their fetch flags are removed.
• [P2] The pin scanner never read shipgate_version:, which action.yml turns into pip install agents-shipgate==<value>. A workflow could name a valid Action ref beside a package version that was never released. Added as a fourth pin pattern, with a negative control that keeps the ref valid and changes only the input. Replaying the reviewer's tracked example now fails: pins shipgate_version input 9.9.9.
• [P2] The vocabulary guard projected each documented set onto the expected values before comparing, so an invented member vanished — needs_a_wizard added to the setup prompt's complete release-decision set still compared equal. Sets are now judged by all of their members, and a literal mixing the two vocabularies fails rather than being skipped by both (the same fail-open from the other side). The projection helper is deleted. Re-scanned every registered surface under the stricter rule: no false positives, the only candidate literals are the four complete copies of the release-decision set.

Negative controls

Each guard has one, and they are not decorative — the first draft's verdict-vocabulary scan matched nothing anywhere in the repository and was replaced:

• test_classifier_rejects_an_unregistered_top_level_entry — a new directory is unclassified.
• test_detector_comparison_reports_a_seeded_divergence — the comparator is not vacuously empty.
• test_pin_scanner_catches_a_seeded_wrong_ref — a wrong emitted Action ref fails.
• test_emitted_ci_workflow_check_catches_a_bad_override — through the emitter, not around it.
• test_contract_floor_check_catches_an_unsatisfiable_floor — on the pairing the surfaces actually shipped.
• test_placeholder_routing_check_catches_a_blanket_instruction — on the wording that actually shipped.
• test_alternation_reader_sees_a_seeded_extra_value, test_verdict_token_parser_rejects_a_seeded_extra_value.

And test_parity_corpus_covers_the_shape_that_produced_the_gap asserts the CLI side of each xfail row, so an xfail cannot be "expected-failing" for a stupid reason.

Offline, with the live check where it already lives

Resolvability is judged against committed metadata: .well-known/agents-shipgate.json gives the source build and the published tag. Discovery and the default static path gain no network calls, and test_resolvability_is_judged_offline pins that by AST. The live check that the claimed tag exists on origin stays in release-tag-consistency — I deliberately did not add a PyPI query, because that job already records why (the index is CDN-cached and would false-fail right after a legitimate release).

PUBLISHED_BUILDS records what each published build implements, committed so a shallow clone is deterministic, and test_published_build_table_matches_the_tag reads it back out of the tag. The suite job now checks out with fetch-depth: 0 / fetch-tags: true so that corroboration actually runs — a check that skips in CI is not a check. test_published_version_metadata_agrees_with_the_source_tree forces the table to be updated at release time.

The harness renders kit templates through the package's own renderer (_render_template), not a copy of it. A parity harness that re-implemented rendering to check for second implementations would be the joke it sounds like.

Deliberately not in this PR

• The The zero-install detector does not read MCP registration sites, so it still reports vendor MCP servers as non-agent projects #485 port. Porting the masking lexer and five idioms into a stdlib-only file is its own increment with its own conformance corpus; The zero-install detector does not read MCP registration sites, so it still reports vendor MCP servers as non-agent projects #485 says so, and its P0 note assigns continuing parity to Ten distribution surfaces, no parity test: #485 proved the zero-install detector and the CLI now disagree #497. This PR gives it a ready-made acceptance test and the fixtures.
• init writes an adopter workflow pinned to a tag that does not exist (56 days and counting) #506's pin repair. It has three candidate designs (published tag / tag SHA / conditional) and an AGENTS_SHIPGATE_WORKFLOW_REF override to weigh. This PR gives it the guard.
• Deep parity for the remaining surfaces. The issue says the inventory follows incrementally and is not a prerequisite. All ten are registered; harness/ gained a real check, policies/ and the MCP server are recorded as claiming nothing, with why.

Surface discipline

Adds no new surface. It constrains surfaces that already shipped and removes two duplicated claims (the plugin/marketplace contract number) and one test exemption. Headline metric: activation — the zero-install path and the emitted workflow are the first two things a cold evaluator runs.

Size

tests/test_distribution_surface_parity.py is ~1,260 lines for 36 tests, which is over the reviewable-PR instinct in #493's proposal. It is one module with one job and heavy per-guard rationale; splitting it would separate each guard from its negative control. Flagging it rather than hiding it.

Verification

Full suite green (exit 0) and ruff check . clean. 11 xfails, all registered gaps: 2 detector rows, 8 pin files, 1 emitted workflow.

Closes #497 (first increment).

🤖 Generated with Claude Code

[pengfei-threemoonslab]

Reviewed commit dbda114d7321c2d9744bdef7104d4c01d482d4cb against c41571f2d11eb5864b487f0ace40a84dfa8f385b.

Recommend fixing the three inline findings before merge: one release/rehearsal CI regression (P1), and two false passes in the new parity guards (P2).

Validation:

• python -m pytest tests/test_distribution_surface_parity.py tests/test_agent_instructions_renderers.py: 247 passed, 11 xfailed.
• Ruff on both changed Python modules: passed. ./shipgate self-check --json: ready: true.
• Reproduced the missing-tag failure in a fresh --depth 1 --no-tags local clone with CI=true, running test_published_build_table_matches_the_tag.
• Mutation checks through the actual guard functions accepted an extra verdict token and a newly tracked workflow with shipgate_version: '9.9.9'. No repository source edits were made.
• The PR's reported CI checks are green; the release-verification workflow is a separate path.

Agents Shipgate verification of the exact base/head: control_state=review_publishable, gate.merge_verdict=human_review_required, release_decision.decision=review_required, permissions.update_pr=true, and permissions.merge=false. Human review remains required for the trust-root changes; the static verifier result is not merge approval.

Posted as a comment review because the authenticated account is also the PR author; the recommendation is to address these findings before merge.

[pengfei-threemoonslab]

[P1] Fetch tags in release verification before requiring them in all CI

This assertion applies to every GitHub Actions job, but only ci.yml's suite checkout was updated. .github/workflows/release-verify.yml:92-96 still checks out the candidate SHA with the default shallow, tagless fetch, then runs the full suite at lines 149-166. Both release.yml and release-rehearsal.yml call that workflow, so they will fail here on the missing v0.15.0 tag even though this PR's CI is green. I reproduced the failure by cloning this head with --depth 1 --no-tags and running CI=true python -m pytest tests/test_distribution_surface_parity.py::test_published_build_table_matches_the_tag. Give the release-verification test checkout the required tags/history as part of this change, and cover that checkout contract too.

[pengfei-threemoonslab]

Confirmed and fixed in 96eefad. You are right that the assertion binds every CI job and only ci.yml's suite checkout was updated — the release path would have gone red on a green PR, which is the worst shape for this kind of regression.

release-verify.yml's tests job now checks out with fetch-depth: 0 and fetch-tags: true, with a comment naming the test that needs them.

On covering the contract: rather than adding the flags and hoping, tests/test_action_metadata.py::test_jobs_running_the_whole_suite_check_out_history_and_tags now asserts it for every job that runs the suite, in any workflow. It classifies a run by whether the pytest invocation has a positional path, so --ignore=tests/test_adapter_static_only.py is correctly not read as scoping the run — that distinction is what separates release-verify.yml's two pytest steps. It detects exactly ci.yml:suite and release-verify.yml:tests today, asserts the detection is non-empty so a reshaped command cannot silently empty it, and both jobs fail the assertion when their fetch flags are removed.

[pengfei-threemoonslab]

[P2] Include the Action's package-version input in the pin scanner

The scanner never reads shipgate_version:, although action.yml:339-342 turns that input into pip install agents-shipgate==<value>. A newly tracked workflow under examples/github-actions/ with uses: ThreeMoonsLab/agents-shipgate@v0.15.0 and shipgate_version: '9.9.9' is selected by _pin_bearing_paths(), but unpublished_pins() returns [] and the registered parity test passes; the adopter's install then fails. The older public-surface test covers an explicit ACTION_PIN_FILES list, so it does not cover newly added workflows. Check the rendered package-version input as well as the Action ref, with a negative control that keeps the ref valid and changes only that input.

[pengfei-threemoonslab]

Confirmed and fixed in 96eefad. shipgate_version: is an install pin wearing a YAML key — action.yml's install step turns it into pip install agents-shipgate==<value> — so a workflow could pass Action-ref resolution and then fail at pip install. Your point about ACTION_PIN_FILES being an explicit list is the reason it matters here: this scanner is path-derived, so it covers a workflow added later.

Added as a fourth entry in PIN_PATTERNS, with the negative control you asked for — the ref stays @v0.15.0 and only the input changes, so a scanner that read only the ref would report nothing.

I also replayed your exact reproduction: created examples/github-actions/99-review-repro.yml with that content, git added it so _surface_files (which reads git ls-files) would see it, and ran the registered test:

FAILED test_executable_pin_resolves_in_a_published_channel[examples/github-actions/99-review-repro.yml]
  examples/github-actions/99-review-repro.yml pins shipgate_version input 9.9.9;
  the newest published release is 0.15.0.

The repro file was removed afterwards. shipgate_version: '<NEW>' in upgrade-shipgate-version.md is not a version and stays unread; every other occurrence in the tree is already '0.15.0'.

[pengfei-threemoonslab]

[P2] Preserve unknown tokens when comparing a documented verdict set

_names(body, vocabulary) projects the literal onto the expected engine values before equality is checked, so an extra, unsupported value disappears. I added "needs_a_wizard" to the actual setup prompt's complete release-decision set and called test_surface_enumerations_match_the_engine_vocabulary() on that mutated text; it passed. This permits a surface to advertise a verdict the engine never emits while the registry claims exact vocabulary parity. Parse all members of a recognized set and compare that full set with the engine vocabulary; add a negative control for a complete valid set plus one unknown value (and reject mixed vocabularies rather than skipping them).

[pengfei-threemoonslab]

Confirmed and fixed in 96eefad. The projection was the bug, and it was doing double duty: it also carried the "which vocabulary is this literal about" disambiguation, which is why a mixed set was skipped by both — the same fail-open from the other side, and you were right to ask for that to be rejected rather than skipped.

vocabulary_set_literals(text, vocabulary) is replaced by documented_vocabulary_sets(text), which parses all members of a candidate literal (a literal is a candidate when two or more members are engine tokens) and returns them unprojected. The test then asserts the member set is exactly MERGE_VERDICTS or exactly RELEASE_DECISIONS; anything else fails, naming both what is in neither vocabulary and what is missing from the nearest match. The projection helper is deleted rather than left around.

Negative controls now cover a complete valid set plus one unknown value, a short set, and a mixed set — plus the two that must keep passing, the complete release-decision set and the complete merge-verdict set, so the overlap on insufficient_evidence/blocked cannot make one read as a short version of the other.

Replayed your reproduction on the real prompt with needs_a_wizard added to the set:

prompts/_review_repro.md spells a verdict set as ['blocked', 'insufficient_evidence',
'needs_a_wizard', 'passed', 'review_required'], which is neither MERGE_VERDICTS … nor
RELEASE_DECISIONS …

The unmutated prompt still passes. I also re-scanned every registered surface under the new rule to be sure the stricter comparison introduces no false positive: the only candidate literals in the tree are the four copies of the setup prompt's release-decision set, all complete.

[pengfei-threemoonslab]

All three findings addressed in 96eefadf; replies are on each thread.

Each was reproduced first and replayed after the fix:

Finding	Repro before	After
P1 release-verify checkout	shallow --depth 1 --no-tags + CI=true fails the corroboration	checkout fetches history and tags; contract asserted for every suite-running job
P2 shipgate_version input	tracked examples/github-actions/99-review-repro.yml passed the parity test	now fails: pins shipgate_version input 9.9.9
P2 projected vocabulary set	needs_a_wizard added to the prompt's set still compared equal	now fails, naming what is in neither vocabulary

The P1 fix is deliberately not just the two flags: tests/test_action_metadata.py::test_jobs_running_the_whole_suite_check_out_history_and_tags now asserts the checkout contract for every job that runs the suite in any workflow, because the finding was precisely that updating one workflow left another behind. It distinguishes --ignore=tests/x from a positional path, which is what separates release-verify.yml's two pytest steps, and asserts it detects a non-empty set of jobs so a reshaped command cannot silently empty it.

Full suite green, ruff check . clean, ./shipgate self-check --json → ready: true, and PR CI is green on the fix commit. Noted that the static verdict is not merge approval and human review remains required for the trust-root changes.

[pengfei-threemoonslab]

Rebased onto main — no new review comments, but #485 and #506 had landed and the branch was CONFLICTING. Now MERGEABLE, full suite green, ruff check . clean, ./shipgate self-check --json → ready: true.

The exemption mechanism got its first real test, and it worked. All three registered gaps were waiting on exactly those two issues. Every xfail(strict=True) row flipped to XPASS, the strict markers failed, and the exemptions had to be removed to get back to green — nobody had to remember. KNOWN_GAPS is empty; the registry keeps a ### Closed table recording what each gap was.

The merge also exposed a duplicate, which is this PR's own subject, so it is removed rather than reconciled. #506 shipped agents_shipgate.published_release and test_adopter_pins_resolve.PIN_SHAPES while this branch carried its own copy of both — down to the shipgate_version input shape we each added independently after your P2 finding. Deleted here and imported instead; the contract-floor regex scan is deleted outright, since #506 made the floor a rendered statement and scanning the older prose would have been a second implementation that was also wrong. The registry now names #506's tests as the proving tests for those claims, which is what a registry is for.

What survives is the half #506's sweep does not reach: pins committed under a registered surface, found by path rather than from an explicit list — the distinction you drew in the P2 comment. test_the_committed_sweep_adds_files_the_emitted_sweep_cannot_see asserts that set is non-empty, so if it ever empties, this scanner is redundant and should go.

One new finding, from importing your stronger ref shape. examples/github-actions/10-check-run-annotations.yml names @main. It is not the #506 defect — @main resolves, and the file's header explains that check_run_policy postdates v0.15.0 — so it is the alternative #497's own rule allows: "a resolvable supported path or an explicit version/contract incompatibility". Enumerated as a declared exception, with a guard that the file really uses that ref and really explains itself, so an unexplained @main elsewhere still fails.

Your P1 fix and main's crossed: main added fetch-tags: true to both ci.yml and release-verify.yml for #506's tag reads. I took main's side — fetch-tags without fetch-depth: 0 is sufficient — and kept the generalized contract test, since those two workflows have now been fixed one at a time twice, once by each of us.