Skip to content

feat!: forge reads native extension keys only (FORGE-1) - #73

Open
Swimburger wants to merge 2 commits into
cloudflare:mainfrom
Swimburger:forge-native-keys-only
Open

Swimburger wants to merge 2 commits into
cloudflare:mainfrom
Swimburger:forge-native-keys-only

Conversation

@Swimburger

@Swimburger Swimburger commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

forge reads the native x-forge-* keys and nothing else, and a new command, forge normalize, rewrites the extensions other tools write (x-fern-*, x-stainless-skip, x-sensitive) into them. FORGE-1.

This is one PR against main: it carries the native vocabulary of RFC 0001 and forge normalize, so it does not depend on any other PR.

What

  • A native vocabulary, kept by src/metadata/ per kind of object, described in rfcs/0001-metadata-resolution-layer.md: x-forge-method-name, x-forge-group-name (a dotted string or an array), x-forge-skip, x-forge-availability, x-forge-model-name, x-forge-property-name, x-forge-parameter-name and x-forge-sensitive.
  • forge normalize rewrites the keys of other tools into those, in place:
    forge normalize 'specs/**/*.json'
    forge normalize --check specs                     # fails if anything would change
    forge normalize --leave-old-extensions spec.json  # keep the old keys beside the native ones
    cat spec.json | forge normalize - | forge - --target ts --out sdk
    It edits the bytes of the document and does not write it again, so whitespace, key order and the text of every number stay as they were, and a diff is only the keys. A native key is never replaced, and a key is rewritten only at the kind of object forge reads its native key at.
  • The tables of other tools' keys are src/normalize/vendor/, per level, and are the only code that names them. A boundary test (tests/boundary.rs) enforces it, and AGENTS.md says that compatibility is sectioned off there.

Two commits:

  1. c012837: all of the above. The engine has no fallback, no tiers and no compat feature.
  2. fe43db8: until the release that makes forge normalize a step to run first, the forge command runs the same rewrite in memory before it parses. Temporary, and only the binary does it.

Breaking

A document written with other tools' keys builds only what its native keys say, unless it has been through forge normalize. The in-memory rewrite keeps the forge command working for now; Rust code that parses a Forge itself does not get it.

Performance

forge normalize on the 28 MB released aggregate, the whole process, best of 20:

first run, rewrites the file in place 26.7 ms
--leave-old-extensions, first run 27.9 ms
already normalized, nothing to write 20.2 ms
--check 23.2 ms
standard input to standard output 23.1 ms

The in-memory rewrite plus the parse, same document, best of 30:

rewrite, raw document (11,714 keys to rewrite) 17.6 ms
rewrite, normalized document (nothing to rewrite) 14.4 ms
parse into a Forge 41.6 ms

The whole forge --target ts run goes from about 252 ms to 262 ms for a normalized document and to 270 ms for a raw one, which is 7%. The rewrite walks the document once with no tree, and checks it to be JSON on a second thread.

Behavior to look at

  • x-sensitive is not read by the engine. forge normalize rewrites it to x-forge-sensitive. It is not called deprecated, since other systems write it, and --leave-old-extensions keeps it for them. @srikrsna: if the engine should keep reading it, that brings back some handling of two keys for one thing.
  • Keys that are left are ordinary extensions, so plugins see them. The default rewrite removes the keys it rewrites.
  • normalize is lenient, and the forge command is not. A key written twice, or a value that cannot be read, is left and warned about by forge normalize. The forge command fails on a key written twice with the error it always gave, duplicate field `x-fern-ignore` at line 1 column 174, and leaves a value that cannot be read for the engine to say.
  • An error forge finds later in the document says the column of the rewritten document, which differs from the one read where a key was renamed earlier on the same line. The line is the same. It goes with the in-memory rewrite.

Verification

On the released aggregate, against the SDK the engine wrote before the change:

  • --target ts writes all 3,243 files, byte for byte, from the raw document (through the in-memory rewrite), from the document after forge normalize in either mode, and through cat spec.json | forge normalize - | forge - --target ts. Without the rewrite, the raw document writes 7 files, so the check can tell.
  • Eight documents with a key written twice, in an operation, an alias, a parameter and a schema, and three with a value that cannot be read, give the same error as before, word for word, line and column included.
  • FORGE_SPEC=openapi.forge.json cargo test --release --test normalize aggregate -- --ignored builds the same model in both modes, with no conflicts.
  • Each commit builds and passes the formatter, clippy (-D warnings) with all features and with none, and its tests (613 and 619). The final tree also compiles for x86_64-pc-windows-msvc.

Follow-up

Remove the in-memory rewrite in the release that makes forge normalize required: revert fe43db8.

…writes the rest

`forge` reads a native `x-forge-*` vocabulary and nothing else. The
extensions other tools write (`x-fern-*`, `x-stainless-skip`,
`x-sensitive`) are ordinary extensions to it, and a new command,
`forge normalize`, rewrites them into the native ones, so the spec that
is produced is itself what gets transformed. FORGE-1.

The native vocabulary: `x-forge-method-name`, `x-forge-group-name` (a
dotted string or an array), `x-forge-skip`, `x-forge-availability`,
`x-forge-model-name`, `x-forge-property-name`,
`x-forge-parameter-name` and `x-forge-sensitive`, kept by
`src/metadata/` per kind of object, alongside `x-forge-hidden`,
`x-forge-require-confirmation`, `x-forge-aliases` and
`x-forge-commands`. The reader asks it for a key and finds one written
twice; there is no fallback and no ranking between keys.

`forge normalize [--leave-old-extensions] [--check] [-v] <INPUT>...`
takes files, folders (every .json file below them, except hidden ones
and those in a node_modules), glob patterns, or - for standard input to
standard output, and rewrites them in place.

- It edits the bytes of the document and does not write it again, so
  whitespace, the order of keys and the text of every number stay as
  they were, and a diff is only the keys. It walks the document once
  with no tree, and checks it to be JSON on a second thread.
- A key is rewritten only at the kind of object `forge` reads its
  native key at, so it never gets a meaning it did not have. The tables
  of other tools' keys are `src/normalize/vendor/`, per level, the only
  code that names them; a boundary test enforces it.
- A native key is never replaced: when a document says the same thing
  twice, the native key is kept, and the other is dropped, or reported
  if it says something else.
- A group path that is an array becomes the names joined by dots, unless
  a name has a dot in it. What no native key says, such as a skip that
  is a list of targets, is left as it is written.
- What it cannot rewrite, a key written twice or a value that cannot be
  read, is left and warned about, with where a parser would stop.
- Files are replaced by writing a new one beside them and renaming it,
  or in place where that is not possible, as for a hard link. A file
  with more than one name is rewritten once, and a link stays a link.
- `--check` writes nothing and fails if anything would change.

RFC 0001 describes the design, and AGENTS.md says that compatibility
with other tools' extensions is sectioned off in `src/normalize/vendor/`.

Breaking: a document written with other tools' keys builds only what its
native keys say until it has been through `forge normalize`.
…or now

Until the release that makes `forge normalize` a step to run first,
`forge` runs the same rewrite on the bytes it reads, before it parses
them, so a document that has not been through `forge normalize` builds
what it did: the released aggregate writes the same 3,243 files of
`--target ts`, byte for byte, without the step. FORGE-1.

It is temporary, and only the binary does it: the engine in `src/` reads
native keys and nothing else.

- It is the default rewrite, so the keys it rewrites are not kept, as
  `forge` did not hand them on to plugins before.
- It says nothing of what it leaves, as `forge` said nothing of the
  document before. A key written twice is an error in the words `forge`
  used when it read those keys itself, down to the line and column:
  "duplicate field `x-fern-sdk-method-name` at line 7 column 30". A value that
  cannot be read is left in place, and `forge` says it as it reads it.
- An error `forge` finds later in the document gives the column of the
  document that was rewritten, which differs from the one that was read
  where a key was renamed earlier on the same line. The line is the same.

Cost, on the 28 MB released aggregate, best of 30: the rewrite is 17.6 ms
when it has 11,714 keys to rewrite and 14.4 ms when it has none, next to
41.6 ms to parse into a `Forge`. The whole `--target ts` run goes from
about 252 ms to 262 ms for a normalized document and to 270 ms for a raw
one, which is 7%.
@Swimburger
Swimburger force-pushed the forge-native-keys-only branch from d0ecf1f to fe43db8 Compare October 28, 2026 01:08
@Swimburger
Swimburger marked this pull request as ready for review October 28, 2026 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant