Skip to content

feat(translator): default to gpt-5.4-mini, behind a model-access error classifier - #136

Merged
SearheiParkhamchuk merged 2 commits into
mainfrom
feat/translator-model-access-error
Sep 11, 2026
Merged

SearheiParkhamchuk merged 2 commits into
mainfrom
feat/translator-model-access-error

Conversation

@SearheiParkhamchuk

Copy link
Copy Markdown
Contributor

Two commits, in this order on purpose: the failure path exists before anything can hit it.

Why the default moves

Measured over 396 live translations, four models, three languages:

Model Usable replies Cost
gpt-4o (today's default) 98 / 99 4.66¢
gpt-5.4-mini 99 / 99 0.86¢

More accurate and roughly five times cheaper.

The policy was published a release ahead of the change: the model option has said since 0.11.0 that the default may move in a minor release and should be pinned when reproducibility matters, and the provider research settled that such a bump ships as a minor feature announced in the release notes rather than as a deprecation. No DEPRECATIONS.md entry.

Probed against the live API before changing anything, rather than assumed:

  • gpt-5.4-mini accepts response_format: json_schema with strict: true — the provider's default envelope works.
  • It accepts temperature. The hypothesis that a newer model would reject sampling parameters — which would have broken every install that sets sampling — is disproved.

Why it cannot ship alone

The one way this hurts an existing install is a key without access to the new model. Before this PR, that failure was invisible:

CLASS  : TransportError
MESSAGE: The translation request failed before a reply could be read…
CAUSE  : 404 The model `…` does not exist or you do not have access to it.
JSON   : {"name":"TransportError","code":"transport"}

cause is installed non-enumerably on purpose — a vendor error can carry an API key — and nothing in the package ever reads it. So the real reason reached no persisted place at all: not the job record, not the log, not the admin.

The classifier

Keys on the vendor's own code, compared exactly — a substring test would catch an unrelated unsupported_model_parameter. One phrase remains as a fallback for a gateway that forwards the text and sets no code (Azure, OpenRouter, a corporate proxy); without it those installs stay on the generic message. Never the bare word model, which such a gateway also echoes inside rate limits.

Unlike the schema-rejection classifier beside it, this one is not gated on structuredOutput — model access has nothing to do with the response envelope.

Getting the reason to the admin

Payload's queue rethrows a handler failure as new TaskError({ message: err.message }) and persists { name, cancelled, message, stack }, where name is its own wrapper class. The error's class, its code and its cause are gone before anything is stored — the message string is the only channel that survives.

So the reason travels as a marker at the head of the message, and the HTTP boundary renders its own catalogue text chosen by that marker. Nothing provider-authored reaches the browser: forging the marker would still only select one of this package's own sentences, and the tail — model name included — stays in the log.

log     : [translator:model-unavailable] The model "…" is not available to this API key. Pass `model` …
browser : The configured translation model is not available to this API key. Set `model` in the provider configuration to one your key can use.

Both routes to a user go through that lookup. withErrorHandler needed it too — it answers synchronous endpoints with e.message directly, so a marked message would otherwise have been shown raw, marker and model name included.

Migration

Anyone who needs the old behaviour pins model: "gpt-4o". The README examples that pass an explicit model are unchanged and still correct; only the table's stated default moved.

Verification

  • 1380 unit tests, 21 new. check-types clean. Lint 58 warnings, 0 errors — identical to main. Declaration build passes.
  • Five mutations, each red on its own case: dropping the code check · reading the code loosely · dropping the phrase fallback · unanchoring the marker · returning the raw message from the synchronous path.
  • Live, on a real key, with NODE_ENV=production: a translation with no model configured returned {"0":"Lies die Dokumentation"}; an inaccessible model produced the actionable message in the log and the catalogue text in the browser.

Known limitation, left alone

classifySchemaRejection, directly above the new check, still classifies by substring and carries the same fragility. It is pre-existing, its codes were not probed, and changing it is its own work.

When an API key cannot reach the configured model, OpenAI answers 404
`model_not_found`. Nothing classified it: the error fell through to
`wrapTransportError`, so the job record and the server log both read "the
translation request failed before a reply could be read" and the real cause
survived only on `cause`, which is installed non-enumerably on purpose and is
never read anywhere in the package. The reason reached no persisted place at
all.

The classifier keys on the vendor's own `code`, compared exactly — a substring
test would catch an unrelated `unsupported_model_parameter`. One phrase
remains as a fallback for a gateway that forwards the text and sets no code
(Azure, OpenRouter, a proxy); without it those installs stay on the generic
message. Never the bare word "model", which such a gateway also echoes inside
rate limits.

Getting the reason in front of an admin needed one more thing. Payload's queue
rethrows a handler failure as `new TaskError({ message: err.message })` and
persists `{ name, cancelled, message, stack }` where `name` is its own wrapper
class, so the error's class and `code` are gone before anything is stored: the
message string is the only channel that survives. A reason therefore travels
as a marker at the head of the message, and the HTTP boundary renders its own
catalogue text chosen by that marker. Nothing provider-authored reaches the
browser — forging the marker would still only select one of our sentences, and
the message tail, model name included, stays in the log.

Both routes to a user now go through that lookup. `withErrorHandler` needed it
too: it answers synchronous endpoints with `e.message` directly, so a marked
message would otherwise have been shown raw.

Verification: 1380 unit tests, 21 new; check-types clean; lint 58 warnings and
0 errors, identical to main. Five mutations, each red on its own case: dropping
the code check, reading the code loosely, dropping the phrase fallback,
unanchoring the marker, and returning the raw message from the sync path.
Measured over 396 live translations across four models and three languages:
gpt-4o returned 98 of 99 usable replies for 4.66 cents, gpt-5.4-mini 99 of 99
for 0.86. More accurate and roughly five times cheaper.

The policy this follows was published a release ahead of the change. The
`model` option has documented since 0.11.0 that the default may move in a
minor release and should be pinned when reproducibility matters, and the
provider research settled that a bump ships as a minor feature announced in
the release notes rather than as a deprecation.

Probed rather than assumed before changing it: gpt-5.4-mini accepts
`response_format: json_schema` with `strict: true`, so the provider's default
envelope works, and it accepts `temperature` — the hypothesis that a newer
model would reject sampling parameters, which would have broken every install
that sets `sampling`, is disproved.

The one way this can hurt an existing install is a key without access to the
new model. That is why the classifier in the preceding commit lands first: the
failure now names the model and points at the `model` option instead of
reaching the admin as "see the server logs".

Anyone who needs the old behaviour pins `model: "gpt-4o"`. The README examples
that pass an explicit model are unchanged and still correct; only the table's
stated default moved.

Verification: 1380 unit tests; check-types clean; lint 58 warnings 0 errors,
identical to main; declaration build passes. Live on a real key with no model
configured: a translation returned, and an inaccessible model produced the
actionable message in the log and the catalogue text in the browser.
@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ideal-cms Ready Ready Preview Sep 11, 2026 9:18am UTC

Request Review

@SearheiParkhamchuk SearheiParkhamchuk self-assigned this Sep 11, 2026
@SearheiParkhamchuk
SearheiParkhamchuk merged commit 22193b0 into main Sep 11, 2026
2 checks passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.12.0 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant