feat(jev): support hosted TypeSafe keys and default to latest

Author: waleedlatif1

apps/docs/content/docs/platform/self-hosting/environment-variables.mdx

@@ -47,6 +47,7 @@ import { Callout } from 'fumadocs-ui/components/callout'
 | `MISTRAL_API_KEY` | Mistral |
 | `XAI_API_KEY_1` | xAI |
 | `KIMI_API_KEY_1` | Moonshot Kimi |
+| `TYPESAFE_API_KEY_1` / `_2` / `_3` | TypeSafe Jev hosted key rotation |
 | `ZAI_API_KEY_1` | Z.ai |
 | `TOGETHER_API_KEY` | Together AI |
 | `FIREWORKS_API_KEY` | Fireworks AI |

apps/docs/content/docs/workflows/blocks/agent.mdx

@@ -33,7 +33,7 @@ Ollama Cloud, OpenRouter, Fireworks, Together AI, Baseten, Ollama, vLLM, and Lit
 
 ### Jev evaluation models
 
-Select `jev-1.13.0`, `jev-latest`, or `jev-preview` from TypeSafe in the Agent model selector, then enter your TypeSafe API key. These models use **State** and **Questions** in place of conversational messages. State accepts text or a reference to a JSON object or array. Questions is a JSON object keyed by the answer names you want:
+Select `jev-latest` from TypeSafe in the Agent model selector. Hosted Sim supplies a key and bills model usage through the normal credit system; workspace or organization BYOK keys override the hosted key without model charges. Self-hosted users enter their TypeSafe key in the block. Use `jev-1.13.0` to pin a version or `jev-preview` to follow preview releases. These models use **State** and **Questions** in place of conversational messages. State accepts text or a reference to a JSON object or array. Questions is a JSON object keyed by the answer names you want:
 
 ```json
 {
@@ -51,9 +51,9 @@ Select `jev-1.13.0`, `jev-latest`, or `jev-preview` from TypeSafe in the Agent m
 }
 ```
 
-Read results from `<agent.answers>`. Each Choice answer includes `choice`, `probabilities`, and `confidence`; each Score answer includes `score`, `legend`, `probabilities`, and `confidence`; each Noul answer includes `noul`, a probability from 0 to 1. `content` contains the same answers as JSON text, and the standard model, token, timing, and cost outputs remain available. Use a Condition block to route on these results.
+Read results from `<agent.answers>` or expand an answer in the reference picker, such as `<agent.answers.route.choice>`. Each Choice answer includes `choice`, `probabilities`, and `confidence`; each Score answer includes `score`, `legend`, `probabilities`, and `confidence`; each Noul answer includes `noul`, a probability from 0 to 1. `content` contains the same answers as JSON text, and the standard model, token, timing, and cost outputs remain available. Use a Condition block to route on these results.
 
-Jev evaluates the supplied state in one request. Chat messages, files, tools, skills, conversation memory, response-format schemas, and chat model fallbacks are hidden for these models. Saved settings return when you switch back to a chat model. Bring your own key; Sim does not provide hosted Jev credits. TypeSafe documents a 64,000-token total request limit and a 32,000-token limit for state plus the longest question. See [TypeSafe's model documentation](https://docs.typesafe.ai/models) and [question formats](https://docs.typesafe.ai/api).
+Jev evaluates the supplied state in one request. Chat messages, files, tools, skills, conversation memory, response-format schemas, and chat model fallbacks are hidden for these models. Saved settings return when you switch back to a chat model. TypeSafe documents a 64,000-token total request limit and a 32,000-token limit for state plus the longest question. See [TypeSafe's model documentation](https://docs.typesafe.ai/models) and [question formats](https://docs.typesafe.ai/api).
 
 ### Files
 
@@ -128,7 +128,7 @@ Some settings live under advanced, or appear only for models that support them:
 - **Max output tokens.** Caps the response length. Defaults to the model's full limit.
 - **Reasoning effort / Thinking level.** For models with extended reasoning, how much the model thinks before answering. Higher is more thorough but slower and costs more tokens.
 - **Prompt caching.** For Anthropic Claude models, reuses the system prompt and tool definitions between runs instead of re-reading them every time. Cached input costs a tenth of the normal rate, but writing the cache costs 1.25x, so leave it off for one-off runs and turn it on when the same agent runs repeatedly. The cache covers a prefix only if it reaches 1,024 tokens (2,048 on Haiku) — below that Anthropic ignores it and nothing changes. Entries expire after five minutes of no use.
-- **API key.** Your key for the chosen provider. Hidden when hosted Sim supplies a key for the selected model. Jev requires your own TypeSafe key.
+- **API key.** Your key for the chosen provider. Hidden when hosted Sim supplies a key for the selected model, including Jev.
 - **Fallback models.** An ordered list of up to five models to try when the request to the selected model fails, whether the provider is overloaded, rate-limited, or down. Sim tries the 2nd choice, then the 3rd, and so on, once each, and `<agent.model>` reports the model that answered. On hosted Sim, hosted models use your workspace's BYOK or platform credentials; local and self-hosted installations may still require a key. A model that needs its own key takes it from a workspace environment variable you pick on the row; a model on the same provider as the selected model reuses the block's key. A stored row key stops applying when its key field is hidden. Providers that require family-specific credentials, such as Vertex, can only be fallbacks for a selected model of the same family. The Auto model cannot be a fallback. A fallback runs with the selected model's settings where its provider accepts them: temperature and max output tokens are clamped to the fallback's limits, and when the fallback has a reasoning effort, thinking level, or verbosity setting that the selected model's value does not fit, the row shows that field so you can pick a value for it; leave it empty and the provider's default applies.
 - **Retry on fail.** Retries the selected model after a failure, up to a maximum number of tries with a wait between them. When its tries run out, the fallback models are tried in order, once each, with no wait before the first of them. A fallback is never retried. See [Retries and fallbacks](#retries-and-fallbacks) for how recorded tool results are reused and when a tool can execute again.
 

apps/sim/app/workspace/[workspaceId]/settings/components/byok/byok.tsx

@@ -39,6 +39,7 @@ import {
   SerperIcon,
   TinyFishIcon,
   TogetherIcon,
+  TypeSafeIcon,
   WizaIcon,
   xAIIcon,
   ZaiIcon,
@@ -130,6 +131,13 @@ const PROVIDERS: (BYOKManagerProvider & { id: BYOKProviderId })[] = [
     description: 'LLM calls',
     placeholder: 'sk-...',
   },
+  {
+    id: 'typesafe',
+    name: 'TypeSafe',
+    icon: TypeSafeIcon,
+    description: 'Jev evaluation models',
+    placeholder: 'Enter your TypeSafe API key',
+  },
   {
     id: 'fireworks',
     name: 'Fireworks',
@@ -352,6 +360,7 @@ const PROVIDER_SECTIONS: BYOKProviderSection[] = [
       'cohere',
       'xai',
       'kimi',
+      'typesafe',
       'fireworks',
       'together',
       'baseten',

apps/sim/blocks/agent-evaluation.test.ts

@@ -1,5 +1,6 @@
 /** @vitest-environment node */
-import { beforeEach, describe, expect, it, vi } from 'vitest'
+import { resetEnvFlagsMock, setEnvFlags } from '@sim/testing'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
 import {
   getEffectiveBlockOutputPaths,
   getEffectiveBlockOutputs,
@@ -19,6 +20,7 @@ const { mockGetBlock } = vi.hoisted(() => ({ mockGetBlock: vi.fn() }))
 vi.mock('@/blocks', () => ({ getBlock: mockGetBlock }))
 
 describe('Agent evaluation configuration', () => {
+  afterEach(resetEnvFlagsMock)
   beforeEach(() => {
     mockGetBlock.mockReturnValue(AgentBlock)
   })
@@ -39,6 +41,12 @@ describe('Agent evaluation configuration', () => {
     }
   })
 
+  it.each([false, true])('shows TypeSafe credentials only when needed, hosted=%s', (hosted) => {
+    setEnvFlags({ isHosted: hosted })
+    const apiKey = AgentBlock.subBlocks.find((field) => field.id === 'apiKey')!
+    expect(evaluateSubBlockCondition(apiKey.condition, { model: 'jev-latest' })).toBe(!hosted)
+  })
+
   it.each(['jev-1.13.0', '<start.model>', '{{MODEL_ID}}'])(
     'exposes answers for %s in downstream selectors',
     (model) => {

apps/sim/lib/api-key/byok.test.ts

@@ -59,6 +59,7 @@ vi.mock('@/stores/providers/store', () => ({
   useProvidersStore: { getState: vi.fn() },
 }))
 
+import { byokProviderIdSchema } from '@/lib/api/contracts/byok-keys'
 import { getApiKeyWithBYOK, getBYOKKey } from '@/lib/api-key/byok'
 import { useProvidersStore } from '@/stores/providers/store'
 
@@ -544,6 +545,89 @@ describe('getApiKeyWithBYOK provider classification', () => {
   })
 })
 
+describe('getApiKeyWithBYOK for TypeSafe', () => {
+  beforeEach(() => {
+    vi.clearAllMocks()
+    resetDbChainMock()
+    mockIsHosted.value = true
+    mockGetHostedModels.mockReturnValue(['jev-latest', 'jev-1.13.0', 'jev-preview'])
+    mockGetRotatingApiKey.mockReturnValue('hosted-typesafe-key')
+    mockDecryptSecret.mockImplementation(async (encrypted: string) => ({
+      decrypted: encrypted.replace('encrypted-', 'decrypted-'),
+    }))
+    mockIsOrganizationBYOKEntitled.mockResolvedValue(true)
+  })
+
+  it('accepts TypeSafe in workspace and organization BYOK contracts', () => {
+    expect(byokProviderIdSchema.parse('typesafe')).toBe('typesafe')
+  })
+
+  it.each(['jev-latest', 'jev-1.13.0', 'jev-preview'])(
+    'resolves the platform pool when %s has no BYOK key',
+    async (model) => {
+      await expect(getApiKeyWithBYOK('typesafe', model, uniqueWorkspaceId())).resolves.toEqual({
+        apiKey: 'hosted-typesafe-key',
+        isBYOK: false,
+      })
+      expect(mockGetRotatingApiKey).toHaveBeenCalledWith('typesafe')
+    }
+  )
+
+  it('prefers the workspace pool without selecting a hosted key', async () => {
+    dbChainMockFns.orderBy.mockResolvedValueOnce([storedKey('workspace-key')])
+    await expect(getApiKeyWithBYOK('typesafe', 'jev-latest', uniqueWorkspaceId())).resolves.toEqual(
+      {
+        apiKey: 'decrypted-workspace-key',
+        isBYOK: true,
+        scope: 'workspace',
+      }
+    )
+    expect(mockGetRotatingApiKey).not.toHaveBeenCalled()
+    expect(mockIsOrganizationBYOKEntitled).not.toHaveBeenCalled()
+  })
+
+  it('inherits an entitled organization pool before using hosted credits', async () => {
+    dbChainMockFns.orderBy
+      .mockResolvedValueOnce([])
+      .mockResolvedValueOnce([storedOrganizationKey(uniqueOrganizationId(), 'organization-key')])
+    await expect(getApiKeyWithBYOK('typesafe', 'jev-latest', uniqueWorkspaceId())).resolves.toEqual(
+      {
+        apiKey: 'decrypted-organization-key',
+        isBYOK: true,
+        scope: 'organization',
+      }
+    )
+    expect(mockGetRotatingApiKey).not.toHaveBeenCalled()
+  })
+
+  it('rejects missing hosted credentials instead of making an unauthenticated request', async () => {
+    mockGetRotatingApiKey.mockImplementation(() => {
+      throw new Error('No configured key')
+    })
+    await expect(getApiKeyWithBYOK('typesafe', 'jev-latest', uniqueWorkspaceId())).rejects.toThrow(
+      'No API key available for typesafe jev-latest'
+    )
+  })
+
+  it('never gives the hosted key to an unlisted model', async () => {
+    await expect(getApiKeyWithBYOK('typesafe', 'jev-custom', uniqueWorkspaceId())).rejects.toThrow(
+      'API key is required'
+    )
+    expect(mockGetRotatingApiKey).not.toHaveBeenCalled()
+  })
+
+  it('requires caller credentials on self-hosted deployments', async () => {
+    mockIsHosted.value = false
+    await expect(
+      getApiKeyWithBYOK('typesafe', 'jev-latest', uniqueWorkspaceId(), 'caller-key')
+    ).resolves.toEqual({ apiKey: 'caller-key', isBYOK: false })
+    await expect(getApiKeyWithBYOK('typesafe', 'jev-latest', uniqueWorkspaceId())).rejects.toThrow(
+      'API key is required'
+    )
+    expect(mockGetRotatingApiKey).not.toHaveBeenCalled()
+  })
+})
+
 describe('getApiKeyWithBYOK for Fireworks', () => {
   const HOSTED_POOL_MODEL = 'fireworks/glm-5.2'
 

apps/sim/lib/api-key/byok.ts

@@ -328,6 +328,7 @@ export async function getApiKeyWithBYOK(
   const isZaiModel = provider === 'zai'
   const isXaiModel = provider === 'xai'
   const isKimiModel = provider === 'kimi'
+  const isTypeSafeModel = provider === 'typesafe'
 
   const byokProviderId = isGeminiModel ? 'google' : (provider as BYOKProviderId)
 
@@ -340,7 +341,8 @@ export async function getApiKeyWithBYOK(
       isMistralModel ||
       isZaiModel ||
       isXaiModel ||
-      isKimiModel)
+      isKimiModel ||
+      isTypeSafeModel)
   ) {
     const hostedModels = getHostedModels()
     const isModelHosted = hostedModels.some((m) => m.toLowerCase() === model.toLowerCase())

apps/sim/lib/api/contracts/byok-keys.ts

@@ -9,6 +9,7 @@ export const byokProviderIdSchema = z.enum([
   'mistral',
   'zai',
   'kimi',
+  'typesafe',
   'xai',
   'fireworks',
   'together',

apps/sim/lib/core/config/env.ts

@@ -261,6 +261,9 @@ export const env = createEnv({
     KIMI_API_KEY_1:                        z.string().min(1).optional(),           // Primary Kimi (Moonshot AI) API key for load balancing
     KIMI_API_KEY_2:                        z.string().min(1).optional(),           // Additional Kimi API key for load balancing
     KIMI_API_KEY_3:                        z.string().min(1).optional(),           // Additional Kimi API key for load balancing
+    TYPESAFE_API_KEY_1:                    z.string().min(1).optional(),
+    TYPESAFE_API_KEY_2:                    z.string().min(1).optional(),
+    TYPESAFE_API_KEY_3:                    z.string().min(1).optional(),
     XAI_API_KEY_1:                         z.string().min(1).optional(),           // Primary xAI API key for load balancing
     XAI_API_KEY_2:                         z.string().min(1).optional(),           // Additional xAI API key for load balancing
     XAI_API_KEY_3:                         z.string().min(1).optional(),           // Additional xAI API key for load balancing

apps/sim/lib/core/utils.test.ts

@@ -24,6 +24,9 @@ beforeAll(() => {
     XAI_API_KEY_1: 'test-xai-key-1',
     XAI_API_KEY_2: 'test-xai-key-2',
     XAI_API_KEY_3: 'test-xai-key-3',
+    TYPESAFE_API_KEY_1: 'test-typesafe-key-1',
+    TYPESAFE_API_KEY_2: 'test-typesafe-key-2',
+    TYPESAFE_API_KEY_3: 'test-typesafe-key-3',
     FIREWORKS_API_KEY_1: 'test-fireworks-key-1',
     FIREWORKS_API_KEY_2: 'test-fireworks-key-2',
     FIREWORKS_API_KEY_3: 'test-fireworks-key-3',
@@ -321,6 +324,9 @@ describe('getInvalidCharacters', () => {
 })
 
 describe('getRotatingApiKey', () => {
+  it.concurrent('rotates the TypeSafe key pool through the shared selector', () => {
+    expect(getRotatingApiKey('typesafe')).toMatch(/^test-typesafe-key-[1-3]$/)
+  })
   it.concurrent('should return OpenAI API key based on current minute', () => {
     const result = getRotatingApiKey('openai')
     expect(result).toMatch(/^test-openai-key-[1-3]$/)

apps/sim/providers/index.test.ts

@@ -2082,13 +2082,14 @@ describe('executeProviderRequest — model level normalization', () => {
 describe('native evaluation provider boundary', () => {
   beforeEach(() => {
     vi.clearAllMocks()
+    envFlagsMockFns.getCostMultiplier.mockReturnValue(2)
     mockExecuteRequest.mockResolvedValue({
       content: '{"passed":true}',
       model: 'jev-1.13.0',
       answers: { passed: true },
       tokens: { input: 100, output: 10, total: 110 },
     })
-    mockGetApiKeyWithBYOK.mockResolvedValue({ apiKey: 'resolved-typesafe-key', isBYOK: false })
+    mockGetApiKeyWithBYOK.mockResolvedValue({ apiKey: 'resolved-typesafe-key', isBYOK: true })
   })
 
   it.each([
@@ -2130,4 +2131,59 @@ describe('native evaluation provider boundary', () => {
       cost: { input: 0, output: 0, total: 0 },
     })
   })
+
+  it.each(['jev-latest', 'jev-1.13.0', 'jev-preview'])(
+    'bills hosted %s using the resolved model price and shared multiplier once',
+    async (model) => {
+      mockGetApiKeyWithBYOK.mockResolvedValue({ apiKey: 'hosted-typesafe-key', isBYOK: false })
+      const result = await executeProviderRequest('typesafe', {
+        model,
+        workspaceId: 'test-workspace',
+        evaluation: {
+          state: 'Task complete',
+          questions: { passed: { type: 'noul', instructions: 'Passed?' } },
+        },
+      })
+      expect(mockExecuteRequest).toHaveBeenCalledWith(
+        expect.objectContaining({ apiKey: 'hosted-typesafe-key', isBYOK: false })
+      )
+      expect(result).toMatchObject({
+        model: 'jev-1.13.0',
+        cost: { input: 0.0000084, output: 0, total: 0.0000084 },
+      })
+    }
+  )
+
+  it.each([false, true])('applies Jev streaming billing consistently, BYOK=%s', async (isBYOK) => {
+    mockGetApiKeyWithBYOK.mockResolvedValue({ apiKey: 'resolved-typesafe-key', isBYOK })
+    const streaming: StreamingExecution = {
+      stream: new ReadableStream(),
+      execution: {
+        success: true,
+        output: {
+          content: '{"passed":{"type":"noul","noul":0.9}}',
+          answers: { passed: { type: 'noul', noul: 0.9 } },
+          model: 'jev-1.13.0',
+          tokens: { input: 100, output: 10, total: 110 },
+          cost: { input: 0.0000042, output: 0, total: 0.0000042 },
+        },
+        logs: [],
+      },
+    }
+    mockExecuteRequest.mockResolvedValue(streaming)
+    await executeProviderRequest('typesafe', {
+      model: 'jev-latest',
+      workspaceId: 'test-workspace',
+      stream: true,
+      evaluation: {
+        state: 'Task complete',
+        questions: { passed: { type: 'noul', instructions: 'Passed?' } },
+      },
+    })
+    expect(streaming.execution.output.cost).toMatchObject({
+      input: isBYOK ? 0 : 0.0000084,
+      output: 0,
+      total: isBYOK ? 0 : 0.0000084,
+    })
+  })
 })