feat(translator): default to gpt-5.4-mini, behind a model-access error classifier - #136
Merged
Merged
Conversation
When an API key cannot reach the configured model, OpenAI answers 404
`model_not_found`. Nothing classified it: the error fell through to
`wrapTransportError`, so the job record and the server log both read "the
translation request failed before a reply could be read" and the real cause
survived only on `cause`, which is installed non-enumerably on purpose and is
never read anywhere in the package. The reason reached no persisted place at
all.
The classifier keys on the vendor's own `code`, compared exactly — a substring
test would catch an unrelated `unsupported_model_parameter`. One phrase
remains as a fallback for a gateway that forwards the text and sets no code
(Azure, OpenRouter, a proxy); without it those installs stay on the generic
message. Never the bare word "model", which such a gateway also echoes inside
rate limits.
Getting the reason in front of an admin needed one more thing. Payload's queue
rethrows a handler failure as `new TaskError({ message: err.message })` and
persists `{ name, cancelled, message, stack }` where `name` is its own wrapper
class, so the error's class and `code` are gone before anything is stored: the
message string is the only channel that survives. A reason therefore travels
as a marker at the head of the message, and the HTTP boundary renders its own
catalogue text chosen by that marker. Nothing provider-authored reaches the
browser — forging the marker would still only select one of our sentences, and
the message tail, model name included, stays in the log.
Both routes to a user now go through that lookup. `withErrorHandler` needed it
too: it answers synchronous endpoints with `e.message` directly, so a marked
message would otherwise have been shown raw.
Verification: 1380 unit tests, 21 new; check-types clean; lint 58 warnings and
0 errors, identical to main. Five mutations, each red on its own case: dropping
the code check, reading the code loosely, dropping the phrase fallback,
unanchoring the marker, and returning the raw message from the sync path.
Measured over 396 live translations across four models and three languages: gpt-4o returned 98 of 99 usable replies for 4.66 cents, gpt-5.4-mini 99 of 99 for 0.86. More accurate and roughly five times cheaper. The policy this follows was published a release ahead of the change. The `model` option has documented since 0.11.0 that the default may move in a minor release and should be pinned when reproducibility matters, and the provider research settled that a bump ships as a minor feature announced in the release notes rather than as a deprecation. Probed rather than assumed before changing it: gpt-5.4-mini accepts `response_format: json_schema` with `strict: true`, so the provider's default envelope works, and it accepts `temperature` — the hypothesis that a newer model would reject sampling parameters, which would have broken every install that sets `sampling`, is disproved. The one way this can hurt an existing install is a key without access to the new model. That is why the classifier in the preceding commit lands first: the failure now names the model and points at the `model` option instead of reaching the admin as "see the server logs". Anyone who needs the old behaviour pins `model: "gpt-4o"`. The README examples that pass an explicit model are unchanged and still correct; only the table's stated default moved. Verification: 1380 unit tests; check-types clean; lint 58 warnings 0 errors, identical to main; declaration build passes. Live on a real key with no model configured: a translation returned, and an inaccessible model produced the actionable message in the log and the catalogue text in the browser.
SearheiParkhamchuk
requested review from
ChiefCreator and
dogfrogfog
as code owners
September 11, 2026 09:15
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
🎉 This PR is included in version 0.12.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, in this order on purpose: the failure path exists before anything can hit it.
Why the default moves
Measured over 396 live translations, four models, three languages:
gpt-4o(today's default)gpt-5.4-miniMore accurate and roughly five times cheaper.
The policy was published a release ahead of the change: the
modeloption has said since 0.11.0 that the default may move in a minor release and should be pinned when reproducibility matters, and the provider research settled that such a bump ships as a minor feature announced in the release notes rather than as a deprecation. NoDEPRECATIONS.mdentry.Probed against the live API before changing anything, rather than assumed:
gpt-5.4-miniacceptsresponse_format: json_schemawithstrict: true— the provider's default envelope works.temperature. The hypothesis that a newer model would reject sampling parameters — which would have broken every install that setssampling— is disproved.Why it cannot ship alone
The one way this hurts an existing install is a key without access to the new model. Before this PR, that failure was invisible:
causeis installed non-enumerably on purpose — a vendor error can carry an API key — and nothing in the package ever reads it. So the real reason reached no persisted place at all: not the job record, not the log, not the admin.The classifier
Keys on the vendor's own
code, compared exactly — a substring test would catch an unrelatedunsupported_model_parameter. One phrase remains as a fallback for a gateway that forwards the text and sets no code (Azure, OpenRouter, a corporate proxy); without it those installs stay on the generic message. Never the bare wordmodel, which such a gateway also echoes inside rate limits.Unlike the schema-rejection classifier beside it, this one is not gated on
structuredOutput— model access has nothing to do with the response envelope.Getting the reason to the admin
Payload's queue rethrows a handler failure as
new TaskError({ message: err.message })and persists{ name, cancelled, message, stack }, wherenameis its own wrapper class. The error's class, itscodeand itscauseare gone before anything is stored — the message string is the only channel that survives.So the reason travels as a marker at the head of the message, and the HTTP boundary renders its own catalogue text chosen by that marker. Nothing provider-authored reaches the browser: forging the marker would still only select one of this package's own sentences, and the tail — model name included — stays in the log.
Both routes to a user go through that lookup.
withErrorHandlerneeded it too — it answers synchronous endpoints withe.messagedirectly, so a marked message would otherwise have been shown raw, marker and model name included.Migration
Anyone who needs the old behaviour pins
model: "gpt-4o". The README examples that pass an explicit model are unchanged and still correct; only the table's stated default moved.Verification
main. Declaration build passes.codecheck · reading the code loosely · dropping the phrase fallback · unanchoring the marker · returning the raw message from the synchronous path.NODE_ENV=production: a translation with nomodelconfigured returned{"0":"Lies die Dokumentation"}; an inaccessible model produced the actionable message in the log and the catalogue text in the browser.Known limitation, left alone
classifySchemaRejection, directly above the new check, still classifies by substring and carries the same fragility. It is pre-existing, its codes were not probed, and changing it is its own work.