Read MCP registration sites in the zero-install detector (#485)

CHANGELOG.md

@@ -40,6 +40,72 @@
   is "names a tag that exists", so it holds on the first commit after the tag
   too.
 
+- **Four lexer defects in the MCP registration reader, fixed in both
+  implementations.** (#485 review) Found reviewing the zero-install port; all
+  four were in the reader #431 shipped, so the port had copied them rather than
+  introduced them. Two invent a tool name, which is the one outcome a reader of
+  a *name* cannot afford, and two lose a whole file's surface:
+
+  - A `${…}` holds code, so a brace inside a string, comment, regex or nested
+    template is not a structural brace. ``const msg = `brace: ${"{"}`;`` left
+    the substitution open and consumed the rest of the file as one unterminated
+    template — every registration after that line gone, and a workspace
+    declaring an MCP dependency reported as "not an agent project" over a brace
+    in a string.
+  - A line break ends a JavaScript initializer only when what follows cannot
+    continue the expression. `static toolName = "safe"` with `+ "_delete"` on
+    the next line published `safe` at `medium` confidence for a tool the server
+    registers as `safe_delete`.
+  - The regex heuristic now resolves the keyword in front of a slash from the
+    *masked* source. Read from the raw text, a comment between `if` and its
+    condition hid the keyword, the slash was read as division, and the pattern
+    was scanned as code — reporting a tool invented out of a regex body, which
+    is precisely what masking exists to make impossible.
+  - A backslash before CRLF is one line continuation, not `\r` plus a line
+    break. The identical file resolved its registration on a Unix checkout and
+    lost it on a Git-for-Windows one.
+
+  Each is an expected-result case in `tests/mcp_idiom_corpus.py`, so both
+  readers are pinned to the corrected behaviour rather than to each other's
+  agreement, and the CRLF sweep now has a continuation case that actually
+  exercises it. The three vendor servers this input exists for are unaffected —
+  61, 114 and 114 tools before and after.
+
+- **The zero-install detector reads MCP registration sites, so it stops
+  telling vendor MCP server maintainers to stop.** (#485) `tools/shipgate-detect.py`
+  is the documented first command run against a repository that has *not*
+  adopted Shipgate — which is every repository #431 was about. #431 taught the
+  installed CLI to read a tool's name out of a TypeScript or Go registration
+  site; the script did not gain it, so the two disagreed on the one question
+  the script exists to answer: `mongodb-js/mongodb-mcp-server` (61 tools),
+  `github/github-mcp-server` (110) and `grafana/mcp-grafana` (114) were agent
+  projects to the CLI and "Stop, not an agent project" to the script. The
+  masking lexer, the five idioms, the path predicate, the dependency gate and
+  the export-precedence rule are now all in the script too, stdlib-only.
+
+  Porting a load-bearing matcher means a second implementation of it, which is
+  this repository's recurring bug class. What makes it affordable is that the
+  two are not allowed to become *different* implementations: every case either
+  reader has ever been asked about now lives once in `tests/mcp_idiom_corpus.py`
+  — every idiom's positive sample, the whole adversarial sweep, the path
+  predicate's cases and both escape grammars — and both readers are driven
+  through all of it, compared site by site including each site's byte span.
+  `samples/mcp_source_only_server` puts the route inside the existing
+  `samples/` parity sweep, nine constructed workspaces pin the branches around
+  it (covering export, partial export, wildcard export, no dependency, no
+  resolved registration, test-only registrations, two registration
+  directories), and `test_framework_vocabulary_names_every_cli_omission` now
+  passes with an empty `known_omissions`.
+
+  One defect surfaced while porting and is fixed in both: with no MCP export in
+  the workspace at all, `_covering_export` returned every resolved name as
+  "uncovered", and the caller renders a shortfall as *"An MCP tool export is
+  also present and does not name N of these registrations"*. A server whose
+  surface exists only as source is the population this input was built for, so
+  that claim about a file that does not exist was published into the adoption
+  evidence for every one of them.
+
+
 - **No corpus case is graded against `insufficient_evidence` any more, and four
   `blocked` cells hold one case instead of two.** (#520, #508) A verdict exists
   to route a change somewhere: `passed` merges, `review_required` hands a human

docs/zero-install.md

@@ -35,21 +35,27 @@ The script's output is a **structural subset** of `agents-shipgate detect --json
   "python_parse_truncated": false,
   "next_action": "agents-shipgate init --workspace .",
   "workspace_signals": {...},
-  "script_version": "0.4.0"
+  "script_version": "0.5.0"
 }
 ```
 
 Like the canonical CLI, the script parse-probes each glob-matched MCP/OpenAPI candidate before suggesting it — a filename match is not a guarantee. A Cursor plugin `mcp.json` is an `mcpServers`-style host config, not an MCP tools-array export; suggesting it would make the next `init --write` → `scan` step fail. Rejected candidates appear under `excluded_sources` (`{type, path, reason}`) instead of `suggested_sources`. The probe is **JSON-only** (stdlib has no YAML parser): a `.json` candidate the adapters would reject is excluded here too, while a `.yaml`/`.yml` OpenAPI spec is always kept as a suggestion (never wrongly dropped). The real-world miss this guards against — `mcpServers`-style host configs — is always JSON, so the probe is exact where it matters.
 
+An MCP server whose tool surface exists **only as TypeScript or Go registration sites** — `mongodb-js/mongodb-mcp-server`, `grafana/mcp-grafana`, `github/github-mcp-server` — is detected here too, and suggested as `{"type": "mcp_server_source", "path": "..."}`. That is 100% of the population this script is pointed at: a repository that has not adopted Shipgate. Until v0.5.0 of the script the reader lived only in the installed CLI, so the documented first command answered "Stop, not an agent project" on exactly the repositories the CLI reported as agent projects with 61, 110 and 114 tools.
+
+Porting it means a second implementation of the load-bearing matcher — the masking lexer and the five registration idioms. It is held to the CLI's answers by a shared conformance corpus rather than by inspection: every positive sample, the whole adversarial sweep, the path predicate and both escape grammars live once in [`tests/mcp_idiom_corpus.py`](https://github.com/ThreeMoonsLab/agents-shipgate/blob/main/tests/mcp_idiom_corpus.py) and are driven through both readers, compared site by site with the byte span of each. Neither reader can change its answer on a case either of them has ever been asked about without the other following.
+
 Like `agents-shipgate detect`, the script silently skips common fixture corpus directories such as `fixtures/`, `_fixtures/`, `__fixtures__/`, `testdata/`, `test_data/`, `test-fixtures/`, `test_fixtures/`, `golden/`, and `goldens/` when they are below the selected workspace. Point `--workspace` directly at a fixture project if you intentionally want to classify that fixture itself.
 
 The script and the canonical CLI are pinned to **structural verdict parity** by [`tests/test_zero_install_detector.py`](https://github.com/ThreeMoonsLab/agents-shipgate/blob/main/tests/test_zero_install_detector.py): same `is_agent_project`, same fired frameworks, same suggested sources, same excluded sources, same Codex plugin candidates, and the same manifest-scope verdict (`agent_scope`, `agent_scope_truncated`, `python_parse_truncated`, plus `agent_project_candidates[]`) for every sample in `samples/`. The scope verdict is pinned because an agent that consults the zero-install path must not adopt a scope the CLI would refuse: on a workspace whose agents live in several self-contained projects, both report `agent_scope: "ambiguous"` and neither recommends initializing the root. `agent_scope_truncated` is pinned for the same reason one step down: when the Python parse stopped at its cap in a workspace holding more than one project root, `agent_project_candidates[]` is a lower bound rather than an enumeration, and a caller that reads a truncated list as complete concludes its own project is not an agent project. `python_parse_truncated` is the wider fact both detectors carry: whether the parse stopped at its cap at all, which is what makes every whole-workspace negative — `is_agent_project: false` included — unsafe to act on. Field-by-field byte parity is not pinned and not promised — the script is not a drop-in replacement for the CLI.
 
 `agent_name_candidates` is the one field pinned byte for byte, including its ranking and each entry's `rationale[]`. It is not a yes/no signal: it names the agent a generated manifest would declare as the reviewed identity, so a script that ranked differently would point you at a different agent than `init` does.
 
+The `mcp_server_source` detection is the one *framework* pinned past its presence — score, confidence, and evidence lines. The looseness the rest of this contract grants was granted to detections scored from many heuristic signals and described in the script's own words; this one has exactly two scoring inputs and its lines are rendered by a ported function. Both halves matter and neither is visible in a verdict: the declared dependency is what carries the label to `medium`, and `"61 tools"` without `"and 3 more this reader cannot name"` is the over-claim the input exists to avoid.
+
 **When to use this:** you're a coding agent (Claude Code, Codex, Cursor) deciding *whether* to propose Shipgate. The script tells you in one fetch + one Python invocation. The full flow (`init`, `scan`, `apply-patches`) requires the actual install.
 
-**Constraints:** Python 3.12+ on the runner. Evidence/reason strings and absolute framework scores are simplified — the verdict is what's pinned, not the prose.
+**Constraints:** Python 3.12+ on the runner. Evidence/reason strings and absolute framework scores are simplified — the verdict is what's pinned, not the prose. `mcp_server_source` is the exception noted above.
 
 The workspace inventory does match the canonical CLI's — `git ls-files` when Git can read the workspace, a contained filesystem walk otherwise. That is a correctness requirement, not a speed one: a `.gitignore`d module is invisible to `init`, so a script that walked it anyway could name an agent `init` will never write. Paths escaping the workspace through a symlink are dropped for the same reason.
 

samples/README.md

@@ -119,6 +119,8 @@ the pointer for the reader, not merely for the test.
 | [`multi_agent_workspace`](multi_agent_workspace/) | Multiple manifests in one workspace. |
 | [`baseline_workflow`](baseline_workflow/) | Baseline adoption before strict CI. |
 | [`large_multi_framework_agent`](large_multi_framework_agent/) | Production-shape retail-ops agent with ~65 unique tools across 6 declared sources, including a reviewed SDK inventory. Exercises the pipeline at scale and pins the CI latency budget. No committed goldens — see the per-sample README. |
+| [`mcp_only_server`](mcp_only_server/) | An MCP server that commits its surface as a `tools/list` export. |
+| [`mcp_source_only_server`](mcp_source_only_server/) | The same server with no export: its tools exist only as TypeScript registration sites, which is the normal state of a vendor MCP server. Detected identically by the CLI and the zero-install script (#485). |
 | [`_anti_patterns`](_anti_patterns/) | Intentionally unsafe or invalid examples for tests and docs. |
 
 ## Direct scans

samples/mcp_source_only_server/README.md

@@ -0,0 +1,70 @@
+# Source-only MCP Tool Server
+
+A minimal MCP server whose tool surface exists **only in its TypeScript
+registration sites**. There is no `tools/list` export to read, which is the
+normal state of a vendor MCP server: the official MongoDB and Grafana servers
+both publish dozens of tools and commit no export at all.
+
+Contrast with [`mcp_only_server`](../mcp_only_server/), which is the same kind
+of server with its surface committed as [`mcp/tools.json`](../mcp_only_server/mcp/tools.json).
+Where both exist, the export wins — it is the server's own published contract,
+it carries the input schemas, and it is read at `high` confidence against this
+route's `medium`.
+
+## What it pins
+
+`detect` reports `is_agent_project: true` and suggests
+`{"type": "mcp_server_source", "path": "src"}` — and **both** detectors do:
+the installed CLI and the zero-install `tools/shipgate-detect.py`, which is the
+documented first command run against a repository that has not adopted
+Shipgate. Until #485 the script had no such reader, so it answered "Stop" on
+exactly the repositories the CLI had just learned to read; this fixture is what
+puts that route inside the parity sweep in
+[`tests/test_zero_install_detector.py`](../../tests/test_zero_install_detector.py).
+
+Two tools are named, one registration is not:
+
+- `support.search_kb` — registered at the call site in
+  [`src/server.ts`](src/server.ts), named by a string literal.
+- `support.drop_ticket_archive` — a tool class in
+  [`src/tools/dropTicketArchive.ts`](src/tools/dropTicketArchive.ts) with a
+  `static toolName` field, plus the sibling `description` and `operationType`
+  literals from the same class body.
+- The `server.registerTool(DropTicketArchiveTool.toolName, …)` call in that
+  same file passes a reference, not a literal. It is reported as
+  **unenumerated** rather than dropped, which is why the evidence line says
+  `1 registration(s) name themselves at runtime and are not enumerated`. A
+  count without that sentence would be an over-claim.
+
+The doc comment in `dropTicketArchive.ts` contains a registration too. It is
+invisible to both readers: comments and string bodies are masked before
+anything is matched, so a documented example can never enter the catalog.
+
+## The whole walk
+
+The route is only worth suggesting if the step after it can act on it, so the
+fixture is checked end to end rather than at `detect` alone:
+
+```bash
+cp -R samples/mcp_source_only_server /tmp/server
+./shipgate detect --workspace /tmp/server --json
+./shipgate init --workspace /tmp/server --write
+./shipgate scan --config /tmp/server/shipgate.yaml
+```
+
+`init` writes `tool_sources: [{id: mcp_src_src, type: mcp_server_source, path:
+src}]`, and `scan` reads both registrations into `report.json`'s `tool_catalog`
+at `medium` confidence, each carrying the file and line it was registered at.
+The registration that names itself at runtime is in `surface_exclusions` as
+`name_not_literal` — accounted for, not dropped, which is what keeps the count
+of two honest.
+
+The terminal then says `Surface: 0 tools` and stops at `insufficient_evidence`,
+and that is the right answer rather than a contradiction: an MCP server has no
+agent object, so nothing binds the catalog's tools to a reviewed surface yet —
+`0/2 catalog tools reachable`. The next step is named
+(`shipgate.yaml#tool_sources[].binding`) and a ready-to-review block is written
+to `suggested-declarations.yaml`. [`mcp_only_server`](../mcp_only_server/),
+whose surface is a committed export, stops in exactly the same place; the
+difference between the two fixtures is how the tools were *found*, not what
+happens after.

samples/mcp_source_only_server/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "@example/support-mcp-server",
+  "version": "0.1.0",
+  "private": true,
+  "description": "A support MCP server whose tool surface exists only in its TypeScript registration sites.",
+  "type": "module",
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.12.0",
+    "zod": "^3.23.8"
+  }
+}

samples/mcp_source_only_server/src/server.ts

@@ -0,0 +1,24 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { z } from "zod";
+
+import { searchKnowledgeBase } from "./handlers.js";
+import { DropTicketArchiveTool } from "./tools/dropTicketArchive.js";
+
+export function createServer(): McpServer {
+  const server = new McpServer({ name: "support", version: "0.1.0" });
+
+  // Registered at the call site. The only thing read out of this file is the
+  // string literal in first-argument position; the schema below is never
+  // evaluated, and the description is not taken from here.
+  server.registerTool(
+    "support.search_kb",
+    {
+      description: "Search support knowledge-base articles by free-text query.",
+      inputSchema: { query: z.string() },
+    },
+    searchKnowledgeBase,
+  );
+
+  DropTicketArchiveTool.register(server);
+  return server;
+}

samples/mcp_source_only_server/src/tools/dropTicketArchive.ts

@@ -0,0 +1,25 @@
+import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+
+type OperationType = "create" | "read" | "update" | "delete";
+
+/**
+ * The class shape the official MongoDB server uses: the tool's identity is a
+ * static field, and the sibling `description` and `operationType` literals are
+ * read from the same class body.
+ *
+ * The registration written in this comment — `static toolName = "example"` —
+ * is invisible to the reader. Comments and string bodies are masked before
+ * anything is matched, so a documented example can never enter the catalog.
+ */
+export class DropTicketArchiveTool {
+  public static readonly toolName: string = "support.drop_ticket_archive";
+  public static operationType: OperationType = "delete";
+  public description = "Delete the archived-ticket collection for a workspace.";
+
+  static register(server: McpServer): void {
+    // Deliberately *not* a literal: the reader reports this registration as
+    // unenumerated rather than dropping it, which is what makes the tool count
+    // in `detect`'s evidence honest about what it could not name.
+    server.registerTool(DropTicketArchiveTool.toolName, {}, () => undefined);
+  }
+}

src/agents_shipgate/cli/discovery/mcp_source.py

@@ -366,8 +366,17 @@ def _covering_export(
     restatement, and only then is withholding it lossless.
     """
 
+    candidates = sorted(exported_source_paths)
+    if not candidates:
+        # No export at all, which is the *common* case here: a server whose
+        # tool surface exists only as source is the population this input
+        # exists for. `names - covered` would be every name, and the caller
+        # renders a non-empty shortfall as "an MCP tool export is also present
+        # and does not name them" — a statement about a file that does not
+        # exist, in the evidence a human reads when deciding whether to adopt.
+        return None, set()
     covered: set[str] = set()
-    for candidate in sorted(exported_source_paths):
+    for candidate in candidates:
         exported = _export_tool_names(workspace, candidate)
         if exported is None:
             continue

src/agents_shipgate/cli/scope_routing.py

@@ -188,7 +188,7 @@ def scope_candidate_actions(
     **A candidate that already carries a manifest routes to ``doctor``, not to
     ``init``.** A nested ``shipgate.yaml`` is itself evidence of a project, so
     adopted directories are candidates too — on this repository's own
-    ``samples/``, 21 of 22 are — and ``init --write`` there exits 2 on a
+    ``samples/``, 23 of 25 are — and ``init --write`` there exits 2 on a
     manifest it will not overwrite while ``expects`` promises a file that
     already exists. ``doctor`` is the command that answers what is actually
     outstanding for an adopted project, and it is the same handoff ``detect``