Skip to content

feat: forge normalize, a command that rewrites other tools' extensions (FORGE-1) - #72

Draft
Swimburger wants to merge 25 commits into
cloudflare:mainfrom
Swimburger:forge-normalize-command
Draft

feat: forge normalize, a command that rewrites other tools' extensions (FORGE-1)#72
Swimburger wants to merge 25 commits into
cloudflare:mainfrom
Swimburger:forge-normalize-command

Conversation

@Swimburger

@Swimburger Swimburger commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Update. #73, stacked on this PR, removes the in-engine fallback, so forge reads the native keys only. It also makes normalize lenient: a key written twice, or a value that cannot be read, is left as it is and warned about, in place of the error described below. The forge command keeps failing on a key written twice, as it always did.

Stacked on #69. This branch contains #69's commits (up to 34a123c) and four of its own. Review the last four commits only: 855118e, 30828f7, cb5f908, d8e6715. It will be rebased onto main once #69 merges.

Addresses the review comment on #69 that vendor compatibility should be a separate command and not something the generator does silently. FORGE-1.

What

forge normalize rewrites the extensions other tools' metadata is written with into the native x-forge-* ones, in place:

forge normalize 'specs/**/*.json'
forge normalize --check specs                     # fails if anything would change
forge normalize --leave-old-extensions spec.json  # keep the old keys beside the native ones
cat spec.json | forge normalize - | forge - --target ts --out sdk

It is meant as a step before generate in CI, and for teams migrating their specs. It is also a library: forge::normalize::normalize.

How it behaves

  • It uses the compatibility layer's own tables, at the same kinds of object that forge reads, so there is no second list of keys, and a key is only rewritten where forge would have read it.
  • It edits bytes, it does not write the document again. Whitespace, key order and the text of every number stay as they were, so a diff of the file is only the keys.
  • Native keys are never replaced. When a document says the same thing twice, the native key wins, and the other is reported as a warning if it says something else.
  • A group path that is an array becomes the names joined by dots, unless a name has a dot in it, which a dotted string cannot say. Those stay arrays.
  • What no native key says is left as written, for example a skip that is a list of targets. A rewrite does not drop information.
  • A key written twice next to a key that is rewritten is an error: forge reads no key that is written twice.
  • Files are replaced atomically (new file beside it, then rename), in place when that is not possible (hard links, read-only folders). A file given under two names is rewritten once. Folders skip hidden directories and node_modules.
  • normalize is a clap subcommand. forge <SPEC> --target ... still works. A document that is itself called normalize is given as ./normalize.

Performance

On the 28 MB released aggregate (release build): about 28 ms for a first run, about 21 ms when it is already normalized, and 24 ms for --check. The document is walked once with no tree, and checked to be JSON on a second thread while it is walked.

Testing

  • Unit and integration tests for every level of object, both modes, conflicts, escapes in keys, CRLF and tabs, comma and whitespace handling when keys are removed, damaged and truncated input, and the CLI (files, folders, globs, stdin and stdout, --check, links, permissions).
  • The released aggregate builds the identical Forge model before and after, in both modes, with no conflicts. That test is #[ignore] and needs FORGE_SPEC:
    FORGE_SPEC=openapi.forge.json cargo test --release --test compat aggregate -- --ignored
  • Each commit builds and passes clippy (-D warnings) with all features and with none.

Not in this PR

  • YAML. JSON only for now; YAML input is an error that says so.
  • Removing the in-engine compatibility fallback. Normalizing first and then removing the fallback is a separate follow-up.
  • Windows has not been run. The in-place fallback for hard links is Unix-only.
  • Consumers that read the old keys themselves need --leave-old-extensions.

@Swimburger
docs: add RFC 0001 for the metadata resolution layer (FORGE-1)
e46cbf1
@Swimburger
docs: fix RFC 0001 dependency direction, vocabulary ownership and ign…
b5cb379 
…ored-value rules
@Swimburger
feat: resolve extensions through a native table with a compat fallbac…
c14688b 
…k (FORGE-1)

Add the metadata resolution layer of RFC 0001.

- src/metadata: the native x-forge-* vocabulary, per kind of object, as
  pure key tables, primitives and value-format descriptors.
- src/compat: the compatibility layer, behind the default-on `compat`
  feature. The Fern and Stainless adapters map their keys onto the same
  primitives; every vendor name lives here.
- src/link/resolve.rs: native first, compat second. A native key's value
  is kept over a compat one's, whatever order they are written in.
- Errors name the key the document wrote.
- Docs and the WIT speak the native keys; the WIT copies are regenerated
  (doc comments only, so forge:plugin stays at 0.3.0).

Output is unchanged: the unmodified Fern-keyed suite passes against it, and
the real aggregate builds the same files and the same model dump.
@Swimburger
test: write the fixtures with the native keys
96c2266 
x-fern-* and x-stainless-skip become their x-forge-* equivalents; a dotted
group string becomes an array. What only the compat layer does is covered
by tests/compat.
@Swimburger
test: guard the boundary of the compatibility layer
f3189e3 
tests/boundary.rs fails, naming the file and line, when an x-fern or
x-stainless name appears in src/, tests/ or packages/forge/ outside
src/compat/ and tests/compat/. `just boundary` runs it.
@Swimburger
docs: record how RFC 0001 was verified
f626473
@Swimburger
perf: turn away other keys early, and keep a resolved key small
abc4896 
- Each table checks its prefix (x-forge-, x-fern-, x-stainless-) before
  matching, and only x- keys are resolved, so a skipped key costs one
  compare instead of a match per table.
- A Hit holds only its field and slot; a key's name comes from the slot,
  in the per-kind name tables, when an error needs it. The reader's
  per-key result is back to 16 bytes, which fits in registers.

Output and the model dump of the aggregate are unchanged.
@Swimburger
chore: add benches/compare.sh, a paired benchmark against a revision
7a9dee1 
Builds REV in a throwaway worktree and runs the parse bench of both in
ABBA order, so drift in the machine's speed cancels, leaving out pairs in
which it changed speed. `just bench-compare [rev] [pairs]` runs it.
@Swimburger
docs: record the measured performance of RFC 0001 and where it went
9f47d32
@Swimburger
feat: rename x-forge-method to x-forge-method-name, add x-forge-sensi…
74a1453 
…tive

- x-forge-method-name, in line with the other -name keys. x-forge-method
  never shipped, so it gets no alias.
- x-forge-sensitive replaces x-sensitive, so every native key begins with
  x-forge-. x-sensitive stays readable as a deprecated native alias: a new
  Tier (native, deprecated, compat) decides which key's value is kept, and
  the alias works with compat off.
- tests/boundary.rs also catches vendor names spelled as identifiers
  (x_fern_...), and the five test names that still had them are renamed.

Output and the model dump of the aggregate are unchanged.
@Swimburger
docs: RFC 0001 takes x-forge-method-name and x-forge-sensitive
afdad0f
@Swimburger
docs: correct RFC 0001 on empty values, the npm release and a stray d…
7ee6854 
…ecision number

- Rule 4 now says per key what an empty or punctuation-only value does:
  names fall back to the wire name; an empty method name or group path is
  no method, and a punctuation-only one is an error.
- The npm release adds no compat module: the engine binary it ships is
  built with the default compat feature.
- "the D4 rule" is resolution rule 2; the D-numbers were never defined.
@Swimburger
docs: record in RFC 0001 that an empty group path should be the root
17af41c 
Today x-forge-group: [] makes no method. The intended meaning is a method
at the root, which needs root methods in the model, the WIT and the
generators, so it is documented as a known gap rather than fixed here.
@Swimburger
test: native keys win inside an alias, and only the winning names are…
dc80bba 
… checked

- In an alias, the native method name, group, availability and ignore
  win over Fern's, in either order.
- A Fern group path or method name that is an error on its own is not
  checked when a native key wins.

Breaking the precedence rule fails all three.
@Swimburger
docs: correct RFC 0001 where it claimed more than the code does
1c1d5ea 
- The boundary goal covers the engine; the docs packages read vendor keys
  themselves and are a non-goal.
- resolve.rs is the only place link calls compat, not the only cfg point.
- Deleting the layer lists every step, and what it leaves behind.
- With compat off, vendor keys become ordinary extensions for plugins.
- The name lookups are a second way vendor key names leave compat, as
  data, for errors.
- Test counts, the mutation result, and performance and memory are
  re-measured on the final code.
@Swimburger
feat: x-forge-group accepts a dotted string, as Cloudflare's specs wr…
aed8392 
…ite groups
@Swimburger
refactor: keep extension tiers local to the reader, and test one key …
e84e0e9 
…per tier
@Swimburger
refactor: address review on RFC 0001
471acdf 
- Rename x-forge-ignore to x-forge-skip, internal names included.
- Rename x-forge-group to x-forge-group-name, to match the other -name
  keys and stay apart from the legacy x-forge-group-info.
- Remove benches/compare.sh and its just recipe; the RFC describes the
  method inline.
- RFC: record why the key is model-name, and that CI compiles but does
  not test the build without compat, on purpose.
@Swimburger
docs: say where vendor compatibility lives, and how to add or remove it
19b3577 
AGENTS.md gets a short section: only src/compat/ names vendor extensions,
it sits behind the compat feature, native keys win, and tests/boundary.rs
enforces it. src/compat/README.md holds the longer how-to, next to the code
it describes and deleted along with it.
@Swimburger
refactor: group extension keys by the level they apply to
a852e33 
Each level (operation or alias, parameter, schema, document) now has its own
module holding its key constants, its table of names and its reader, in the
native vocabulary and in each adapter. A key constant is no longer in scope
at a level it does not belong to. Tests check that every key is read at
exactly the levels that list it.

Also drop the key tables from the adapters' module docs: the code is the
source of truth for the keys.
@Swimburger
docs: trim RFC 0001 to the essentials
34a123c 
Drop the status header, the known gaps and follow-ups sections, the test plan
and the release shape, and shorten the rest. The vocabulary, the resolution
rules, the architecture and the alternatives considered stay.
@Swimburger
test: a group path is split on dots only when it is a string
855118e 
A group path written as a string is names separated by dots, and one
written as an array is the names themselves, which are not split, so
["a.b", "c"] is two names and "a.b.c" is three.

`forge normalize` writes an array as a dotted string only when it says
the same thing, so it depends on the two readings staying as they are.
@Swimburger
feat: forge::normalize rewrites other tools' extensions to the native…
30828f7 
… ones

Rewrites the extensions other tools' metadata is written with into the
native x-forge-* ones, in a JSON document, using the keys the
compatibility layer reads: a key is only rewritten where `forge` would
have read it, so it never gets a meaning it did not have. FORGE-1.

The document is edited as bytes and not written again, so everything
but the keys that change, such as the text of every number, the
whitespace and the order of the keys, stays as it was. It is walked
once, without a tree: 28 MB in about 20 ms.

- Native keys are never replaced. When a document says the same thing
  twice, the native key wins, and the other is reported if it says
  something else.
- A group path that is an array becomes the names joined by dots, unless
  a name has a dot in it, which a dotted string cannot say.
- What no native key says, such as a skip that is a list of targets, is
  left as it is written.
- A key written twice next to a key that is rewritten is an error, as
  `forge` reads no key that is written twice.
- The document is checked to be JSON beside the walk, on another thread
  when it is long, so that what is written does not have to be.
- `leave_old_extensions` keeps each key that is rewritten, beside the
  native key that replaces it.

The native key of each field is named by the per-level modules of
`metadata`. The feature is `normalize`, on by default, and needs
`compat`.
@Swimburger
feat: forge normalize command
cb5f908 
`forge normalize [--leave-old-extensions] [--check] [-v] <INPUT>...`
rewrites other tools' extensions into the native ones, in place. FORGE-1.

An input is a file, a folder (every .json file below it, except hidden
ones and those in a node_modules), a glob pattern, which is taken as it
is written, or - for standard input to standard output.

- One line for each file that is rewritten, and a summary. `-v` adds how
  many times each key was rewritten. Conflicts are warnings.
- `--check` writes nothing and fails if anything would change, to check
  that documents are normalized.
- Files are rewritten by writing a new file beside them and renaming it,
  so that none is left half written, and in place where that is not
  possible, as for a hard link. A file with more than one name is
  rewritten once, and a link stays a link.
- A file that is not valid JSON, is read-only, or is YAML is an error and
  is left as it is.

`normalize` is a subcommand of clap's: `args_conflicts_with_subcommands`
lets `forge <SPEC> --target ...` and `forge normalize ...` be told
apart. A document that is itself called `normalize` is given as
`./normalize`.
@Swimburger
docs: document forge normalize
d8e6715 
The package README says how to use it and what it leaves alone, and the
compatibility layer's README says how it is built on the adapter
tables, what it does that is not a key becoming its native key, and what
to delete to remove it with the layer.
This was referenced Oct 28, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Swimburger