What I did
- Open Settings, then AI providers, then Add provider, then OpenAI.
- Paste a key that is not valid.
- Press Test.
What I expected
The drawer tells me the key was rejected, and shows no model list. There is no key, so there is no list of models the key can reach.
What happens instead
The drawer shows the rejection, and then shows a full model catalog anyway:
- The status line reads "OpenAI rejected this key (401) Nothing has been saved." The two sentences run together. There is no full stop between
(401) and Nothing.
- Under it, "Active models — 3 of 34", with a searchable list of 34 models, and three of them already ticked:
gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, each marked "recommended".
- The helper text under the status line reads: "Test checks the key with OpenAI and fetches its model list. Nothing is saved until Done." That is not what happened. OpenAI rejected the key, so nothing was fetched, but a list appeared.
I then repeated the same steps with a key that OpenAI accepts. I got the same 34 models with the same three ticked. So the list told me nothing about either key.
Done is correctly disabled after a rejected key. The save refusal works. Only the list and the copy are wrong.
Observed environment: staging, v0.114.4.
Screenshot: j2-invalid-key-test-result.png.
Suggested implementation path
What is really going on
The model list is not hardcoded. A live fetch already exists and already works. POST /providers/probe returns both a credential verdict and a discovered model list, and the drawer does render the discovered list when it arrives. web/packages/agenta-entity-ui/src/secretProvider/ProviderConnectionCard.tsx:184 and :193-196:
const discovered = probe?.discovery.status === "fetched"
const available = useMemo(
() => (discovered ? (probe?.discovery.models ?? []) : catalog.models),
[discovered, probe, catalog.models],
)The bug is the silent fallback on the other branch. Whenever discovery.status is anything but fetched, the drawer quietly substitutes Agenta's shipped catalog and says nothing. That catalog is the 34 OpenAI ids in sdks/python/agenta/sdk/utils/assets.py:93-126, and the three ticked defaults are PROVIDER_DEFAULT_MODELS["openai"] at sdks/python/agenta/sdk/agents/capabilities.py:146-151. They reach the browser through the harness capability catalog, and are combined by providerModelCatalog at web/packages/agenta-entities/src/secret/core/connections.ts:406-435.
A rejected key always lands on that branch, because the probe never gets far enough to list models. So the user is shown a catalog that is unrelated to the key they just typed, with no sign that it is a fallback.
There is a second, quieter case worth catching in the same change. When the credential verdict is unknown, the card still paints the green success dot, because credentialFailed only covers invalid and a transport failure (ProviderConnectionCard.tsx:352), and doneState still enables Save (connections.ts:601-606). A probe that could not reach the provider therefore reads as a pass, with the shipped catalog under it. That is the most likely reason the valid key and the rejected key looked identical in this report.
The change
Three parts. All are small, and all sit in two files plus their tests.
1. Say when the list is not the key's list. In ProviderConnectionCard.tsx:193-196, keep the fallback, but pass the discovery status down. In ActiveModelsSection.tsx:180-212, print a line when the status is not fetched, for example: "Showing Agenta's shipped catalog. OpenAI's own list was not fetched." Add the prop to ActiveModelsSectionProps at ActiveModelsSection.tsx:30-42. About 15 lines across the two files.
Stronger option, if the team prefers it: hide the model section entirely while the credential verdict is invalid. There is nothing useful to choose from at that moment, and Done is disabled anyway. This is fewer lines than the note, but it removes a section the user may have already scrolled to. Recommendation: ship the note first, because it also covers the unknown and failed cases, which hiding does not.
2. Stop painting unknown with the success dot. ProviderConnectionCard.tsx:352 and :470-474. A third, neutral state is more honest than either the green dot or the red warning. About 8 lines.
3. Fix the run-together sentences. This is the most contained part and can go first. credentialStatusLine strips the trailing full stop unconditionally, at web/packages/agenta-entities/src/secret/core/cardCopy.ts:20-24:
const verdict = message.trim().replace(/\.$/, "")
if (modelCount === null) return verdict
return `${verdict} · ${modelCount} ${modelCount === 1 ? "model" : "models"} fetched`The strip exists so that the middle dot separator does not read past a full stop. But when modelCount is null, no separator follows, and the caller appends a second sentence to the stripped string. ProviderConnectionCard.tsx:475-478:
<span>
{statusLine}
{credentialFailed ? " Nothing has been saved." : null}
</span>With the API message "OpenAI rejected this key (401)." the result is the string in the screenshot.
Fix: only strip the full stop when the · N models fetched suffix is about to be appended. Move the strip inside the modelCount !== null branch. About 4 lines. Then update the assertion that pins the current behaviour, at web/packages/agenta-entities/tests/unit/provider-card-copy.test.ts:36-40.
The same defect hits the transport path, where probeFailureMessage (connections.ts:258-271) also loses its full stop before " Nothing has been saved.". One fix covers both.
What does not need work
The helper text at cardCopy.ts:65-68 is accurate for the path it describes. Leave the copy alone and fix the silent fallback instead. If part 1 is rejected, then the helper text must be softened as the fallback fix, but that is the weaker outcome.
No backend work is needed. The probe endpoint, its adapters, and its DTOs already exist and already return a real per-key model list.
Why this is a good first issue
Part 3 alone is a complete, useful change: about 4 lines of source and 2 lines of test, in one function, with the failing string visible in the screenshot. Parts 1 and 2 are each contained to one component, with the exact lines named above.
What I did
What I expected
The drawer tells me the key was rejected, and shows no model list. There is no key, so there is no list of models the key can reach.
What happens instead
The drawer shows the rejection, and then shows a full model catalog anyway:
(401)andNothing.gpt-5.6-sol,gpt-5.6-terra,gpt-5.6-luna, each marked "recommended".I then repeated the same steps with a key that OpenAI accepts. I got the same 34 models with the same three ticked. So the list told me nothing about either key.
Done is correctly disabled after a rejected key. The save refusal works. Only the list and the copy are wrong.
Observed environment: staging, v0.114.4.
Screenshot:
j2-invalid-key-test-result.png.Suggested implementation path
What is really going on
The model list is not hardcoded. A live fetch already exists and already works.
POST /providers/probereturns both a credential verdict and a discovered model list, and the drawer does render the discovered list when it arrives.web/packages/agenta-entity-ui/src/secretProvider/ProviderConnectionCard.tsx:184and:193-196:The bug is the silent fallback on the other branch. Whenever
discovery.statusis anything butfetched, the drawer quietly substitutes Agenta's shipped catalog and says nothing. That catalog is the 34 OpenAI ids insdks/python/agenta/sdk/utils/assets.py:93-126, and the three ticked defaults arePROVIDER_DEFAULT_MODELS["openai"]atsdks/python/agenta/sdk/agents/capabilities.py:146-151. They reach the browser through the harness capability catalog, and are combined byproviderModelCatalogatweb/packages/agenta-entities/src/secret/core/connections.ts:406-435.A rejected key always lands on that branch, because the probe never gets far enough to list models. So the user is shown a catalog that is unrelated to the key they just typed, with no sign that it is a fallback.
There is a second, quieter case worth catching in the same change. When the credential verdict is
unknown, the card still paints the green success dot, becausecredentialFailedonly coversinvalidand a transport failure (ProviderConnectionCard.tsx:352), anddoneStatestill enables Save (connections.ts:601-606). A probe that could not reach the provider therefore reads as a pass, with the shipped catalog under it. That is the most likely reason the valid key and the rejected key looked identical in this report.The change
Three parts. All are small, and all sit in two files plus their tests.
1. Say when the list is not the key's list. In
ProviderConnectionCard.tsx:193-196, keep the fallback, but pass the discovery status down. InActiveModelsSection.tsx:180-212, print a line when the status is notfetched, for example: "Showing Agenta's shipped catalog. OpenAI's own list was not fetched." Add the prop toActiveModelsSectionPropsatActiveModelsSection.tsx:30-42. About 15 lines across the two files.Stronger option, if the team prefers it: hide the model section entirely while the credential verdict is
invalid. There is nothing useful to choose from at that moment, and Done is disabled anyway. This is fewer lines than the note, but it removes a section the user may have already scrolled to. Recommendation: ship the note first, because it also covers theunknownandfailedcases, which hiding does not.2. Stop painting
unknownwith the success dot.ProviderConnectionCard.tsx:352and:470-474. A third, neutral state is more honest than either the green dot or the red warning. About 8 lines.3. Fix the run-together sentences. This is the most contained part and can go first.
credentialStatusLinestrips the trailing full stop unconditionally, atweb/packages/agenta-entities/src/secret/core/cardCopy.ts:20-24:The strip exists so that the middle dot separator does not read past a full stop. But when
modelCountisnull, no separator follows, and the caller appends a second sentence to the stripped string.ProviderConnectionCard.tsx:475-478:With the API message
"OpenAI rejected this key (401)."the result is the string in the screenshot.Fix: only strip the full stop when the
· N models fetchedsuffix is about to be appended. Move the strip inside themodelCount !== nullbranch. About 4 lines. Then update the assertion that pins the current behaviour, atweb/packages/agenta-entities/tests/unit/provider-card-copy.test.ts:36-40.The same defect hits the transport path, where
probeFailureMessage(connections.ts:258-271) also loses its full stop before " Nothing has been saved.". One fix covers both.What does not need work
The helper text at
cardCopy.ts:65-68is accurate for the path it describes. Leave the copy alone and fix the silent fallback instead. If part 1 is rejected, then the helper text must be softened as the fallback fix, but that is the weaker outcome.No backend work is needed. The probe endpoint, its adapters, and its DTOs already exist and already return a real per-key model list.
Why this is a good first issue
Part 3 alone is a complete, useful change: about 4 lines of source and 2 lines of test, in one function, with the failing string visible in the screenshot. Parts 1 and 2 are each contained to one component, with the exact lines named above.