[Feature] 实验性探索：使用 Jev 作为独立的权限审核后端

[xingkaixin]

你需要什么？

当前「助手代审」使用生成式大模型评估工具操作，返回自动放行、询问用户或阻止执行。审核使用的 assistantModel 同时承担上下文压缩，因此不能直接将其替换为不擅长文本生成的 Jev。

本需求探索新增独立的 Jev 权限审核选项。该能力定位为实验性功能，不默认启用，是否采用取决于实际评估结果。

目标

• 评估 Jev 是否适合承担具体工具操作的权限审核。
• 保持现有权限交互流程和审核结果类型兼容。
• 将权限审核后端的选择与用于上下文压缩的助手模型分开。
• 对比现有审核方式的判断质量、延迟和成本，形成采用或不采用的结论。

“行为兼容”指保留相同的权限处理方式，不承诺 Jev 与现有模型对每个操作做出完全相同的判断。

非目标

• 替换通用助手模型或上下文压缩模型。
• 扩大需要审核的工具范围，或放宽现有权限规则。
• 绕过必须由用户确认的操作。
• 让审核模型自主执行工具、读取文件或调查外部环境。
• 要求 Jev 生成与原审核助手相同的自由文本解释。
• 在缺少评估证据的情况下默认启用。

预期行为

用户可以显式选择实验性的 Jev 权限审核后端。未选择时，保持现有行为。

审核时，将具体工具操作、参数、权限信息和相关对话作为输入，由 Jev 提供结构化判断，再由本地代码映射为现有审核结果：

情况	预期处理
操作符合自动放行条件，且判断达到经过验证的置信度要求	自动放行
风险较高、授权不清楚、信息不足或判断不确定	询问用户
按审核策略判定为严重风险	阻止执行
请求失败、超时或响应无效	询问用户
操作明确要求用户确认	保留人工确认，不由 Jev 覆盖

其他行为约束：

• 审核结果必须绑定被审核的具体操作和参数，不能复用于其他操作。
• 保留现有代码对 high 风险询问用户、critical 风险阻止执行的约束。
• 原助手模型继续承担上下文压缩。
• 审核说明可由结构化分类映射为固定文案，不伪装成模型生成的详细解释。
• 置信度阈值通过场景评估确定，不将高置信度直接视为安全保证。

已知限制与待验证项

• 提示注入：官方说明 Jev 1.13 的判断可能受输入中的对抗性内容影响，需要覆盖工具参数、工具输出和引用内容中的注入。
• 复杂命令：涉及嵌套脚本、间接数据流或多步推理时，不能假定其判断能力与现有生成式模型等价。
• 中文授权：需要验证中文指令、否定表达、授权撤回和多轮授权场景。
• 上下文完整性：当前审核使用最近 8 条消息，每条正文最多 2,000 字符。更早的授权或限制可能缺失，替换模型本身不能解决这一问题。
• 数据发送范围：接入云端审核会向 TypeSafe 发送审核输入，需要明确必要字段及敏感信息处理方式。
• 模型版本：评估需记录具体版本，避免模型别名更新后无法复现结果。

评估与完成标准

使用脱敏的真实权限请求和必要的边界案例，与现有审核方式进行对比。以明确的权限策略和人工标注作为判断依据，原模型的输出仅作为对照。

重点观察：

• 危险操作被错误放行的比例。
• 正常操作被错误阻止或不必要地转交人工的比例。
• 中文授权、提示注入和复杂命令场景的表现。
• 自动放行率、P50/P95 延迟、单次审核成本。
• 失败、超时和不确定时是否正确回到人工确认。

完成标准是形成有证据的结论：适合使用、仅适合有限场景，或暂不适合。成功调用 API 本身不代表实验通过。

参考资料

• Jev 简介与结构化判断设计
• HTTP API：输入与输出格式
• 模型规格、价格、语言支持和数据处理说明
• Jev 1.13 已知限制
• 置信度的含义与使用方式
• Guardrails 示例
• JavaScript / TypeScript SDK

参考文档查阅日期：2026-09-18。上述能力说明基于当时文档中的 Jev 1.13，后续评估应重新核对版本和限制。

[xingkaixin]

补充可选的 API 接入渠道：除 TypeSafe 官方 API 外，目前 Vercel AI Gateway、Cloudflare 和 OpenRouter 也已提供 Jev 的模型页面及相关 API 接入入口，可纳入本实验的接入方案评估。

平台	模型标识	参考链接
Vercel AI Gateway	typesafe-ai/jev	Jev 模型页面及调用示例
Cloudflare	typesafe/jev	Jev 模型文档及 API 示例
OpenRouter	typesafe/jev-1.13	Jev 1.13 模型页面

后续评估可比较各渠道的请求/响应格式、模型版本、鉴权方式、延迟、价格和限流。不同平台的接口封装可能不同，不能仅因已有该平台的聊天模型接入，就假定 Jev 可以直接复用现有聊天调用链路。

以上为接入渠道补充，尚未进行实际 API 调用验证；不改变本 issue 的实验性定位和权限审核范围。

[zerob13]

[Feature] Experimental exploration: use Jev as a standalone permission review backend

What is needed

The current assistant-based review ("Approve for me") uses a generative LLM to evaluate tool operations and returns one of: auto-approve, ask the user, or block. The assistantModel that performs this review also handles context compression, so it cannot be replaced outright by Jev, which is not built for text generation.

This issue explores adding a standalone Jev permission review option. It is positioned as an experimental feature, off by default; whether to adopt it depends on evaluation results.

Goals

• Evaluate whether Jev is fit for reviewing concrete tool operations.
• Keep the existing permission flow and review result types compatible.
• Separate the choice of permission review backend from the assistant model used for context compression.
• Compare judgment quality, latency, and cost against the current approach, and reach a clear adopt / do-not-adopt conclusion.

"Behavior compatible" means the same permission handling is preserved; it does not promise that Jev will judge every operation identically to the current model.

Non-goals

• Replacing the general assistant model or the context compression model.
• Expanding the set of tools that require review, or relaxing existing permission rules.
• Bypassing operations that must be confirmed by the user.
• Letting the review model itself execute tools, read files, or probe the external environment.
• Requiring Jev to produce the same free-text explanations as the current review assistant.
• Enabling by default in the absence of evaluation evidence.

Expected behavior

The user can explicitly opt in to the experimental Jev review backend. Without opt-in, current behavior is unchanged.

At review time, the tool operation, its parameters, permission info, and the relevant conversation are sent as input; Jev returns a structured judgment, which local code maps to the existing review results:

Case	Expected handling
Operation meets auto-approve conditions and the judgment reaches a validated confidence threshold	Auto-approve
Higher risk, unclear authorization, insufficient info, or uncertain judgment	Ask the user
Severe risk per the review policy	Block execution
Request failure, timeout, or invalid response	Ask the user
Operation explicitly requires user confirmation	Keep manual confirmation; Jev cannot override it

Other constraints:

• A review result must be bound to the specific operation and parameters it reviewed; it cannot be reused for other operations.
• Keep the existing constraints: high risk asks the user, critical risk blocks execution.
• The existing assistant model continues to handle context compression.
• Review explanations may be fixed copy mapped from structured classifications, not presented as model-generated detailed explanations.
• Confidence thresholds are determined through scenario evaluation; high confidence alone is not treated as a safety guarantee.

Known limits and open questions

• Prompt injection: official docs note that Jev 1.13's judgment can be affected by adversarial content in the input. Injections in tool parameters, tool outputs, and quoted content must be covered.
• Complex commands: for nested scripts, indirect data flow, or multi-step reasoning, do not assume parity with the current generative model.
• Chinese authorization: verify Chinese instructions, negation, revoked authorization, and multi-turn authorization scenarios.
• Context completeness: the current review uses the last 8 messages, each body truncated to at most 2,000 characters. Earlier grants or restrictions may be missing; swapping the model does not fix this.
• Data sent: a cloud review sends review input to TypeSafe; the required fields and how sensitive info is handled must be defined.
• Model version: record the exact version in the evaluation, since model alias updates could make results unreproducible.

Evaluation and completion criteria

Compare against the current approach using sanitized real permission requests plus the necessary edge cases, with an explicit permission policy and human labels as ground truth; the original model's output serves only as a reference.

Focus on:

• Rate of dangerous operations wrongly auto-approved.
• Rate of normal operations wrongly blocked or unnecessarily escalated to the user.
• Performance on Chinese authorization, prompt injection, and complex command scenarios.
• Auto-approve rate, P50/P95 latency, and cost per review.
• Whether failures, timeouts, and uncertainty correctly fall back to manual confirmation.

Completion means an evidence-backed conclusion: suitable, suitable only for limited scenarios, or not yet suitable. A successful API call alone does not mean the experiment passed.

References

• Jev introduction and structured judgment design
• HTTP API: input and output formats
• Model specs, pricing, language support, and data handling
• Jev 1.13 known limitations
• Confidence semantics and usage
• Guardrails examples
• JavaScript / TypeScript SDK

References were reviewed on 2026-09-18; the capability notes above are based on Jev 1.13 as documented at that time. Later evaluations should re-check the version and limitations.

---

Author's follow-up (comment): Besides the official TypeSafe API, Vercel AI Gateway, Cloudflare, and OpenRouter now also list Jev models with API access, and can be included in the evaluation:

Platform	Model ID	Reference
Vercel AI Gateway	typesafe-ai/jev	Jev model page and examples
Cloudflare	typesafe/jev	Jev model docs and API examples
OpenRouter	typesafe/jev-1.13	Jev 1.13 model page

The follow-up comparison should cover request/response formats, model versions, auth, latency, pricing, and rate limits across channels. Platform wrappers differ, so an existing chat-model integration on a platform cannot be assumed to work for Jev directly. No actual API calls have been made yet; this does not change the issue's experimental positioning or its review scope.

_{Automatically translated by DimAgent}

[zerob13]

我的jev还没来得及去申请，方便的话要不直接来个pr吧 😁

[xingkaixin]

typesafe 目前应该是申请就有。
我对这个项目架构不熟悉，Vue 技术栈完全没接触过，我只能对功能结果进行验证，无法对代码质量做评估。虽然是实验性功能，但考虑到项目的大量受众，我提 PR ，我无法对 PR 质量做保证，所以还是让熟悉的人来做吧。👀

[zhangmo8]

Status: implementation landed in #2335; feasibility still unverified

#2335 implements the buildable half of this issue. It is approved and green, but it does not
close this issue, because the completion criterion here is an evidence-based conclusion and no
evaluation has been run. This comment records exactly what is and is not done, and what the
documentation implies about whether the remaining part is achievable.

Requirement status

Requirement	Status
Goal 2 — keep the permission flow and result types compatible	Done. Result type unchanged, flow unchanged, opt-in.
Goal 3 — separate the review backend from the assistant model used for compaction	Done. New judgmentModel slot, read only by the reviewer; compaction and the other three assistantModel readers untouched.
Goal 1 / Goal 4 — evaluate quality, latency, cost and reach a conclusion	Not done.
Non-goals (6)	All respected. Not default-enabled, no tool scope widened, no user-confirmation bypass, no tool execution by the model, no free-text explanation required.
Expected behaviour table (5 rows)	All implemented. The last row — an action that requires user confirmation is not overridden — is enforced upstream: the reviewer is only consulted in auto_approve mode.
"Thresholds determined by scenario evaluation"	Half done. The thresholds exist but are placeholders. The other half — never treating high confidence as a safety guarantee — holds: confidence is ANDed with risk, authorization and injection signals.
Known limitation: prompt injection	Mechanism built (an injection question, tool results retained, head+tail truncation). Verification not done.
Known limitation: complex commands	Not tested. No equivalence is claimed.
Known limitation: Chinese authorization	Not verified at all.
Known limitation: context completeness	Input filtered, but the root cause — earlier authorization falling outside the window — is unsolved, as this issue already notes.
Known limitation: data sending scope	Partial. The state is filtered to the fields the questions need, and the settings UI now discloses that tool arguments and the recent conversation go to the configured service. The exact field list and sensitive-data handling are not documented.
Known limitation: model version	Mechanism in place. The response's versioned id is logged, and the built-in catalog pins jev-1.13.0.

Two notes on the implementation that matter for the evaluation:

• The judgment path caps auto-allow at low risk, while the generative reviewer allows low and
  medium. That difference is deliberate and now an explicit threshold, but it means the two paths
  do not share a policy. An evaluation that counts interruptions is therefore not measuring the
  model alone; the cap should be aligned before drawing conclusions about judgment quality.
• Current placeholder thresholds: autoAllowMaxRiskLevel: low, autoAllowMinAuthorization: 0.8,
  autoAllowMaxInjectionPressure: 0.2, autoAllowMinRiskConfidence: 0.6.

Theoretical feasibility: what the documentation supports, and what it does not

Splitting the requirements into model-independent and model-dependent parts gives opposite answers.

Model-independent (integration) — achievable, and already demonstrated. The behaviour table,
action binding, failure fallback and user-confirmation precedence are integration concerns, and
Jev's wire shape (Choice/Noul with probabilities) maps onto them directly.

Model-dependent — three dimensions look favourable, three are in doubt.

Favourable, with evidence:

• Latency and cost. Input-only pricing at $0.042/Mtok with output free, 1200 req/min, and a
  documented measurement of 13 questions answered in a single call in 0.27 s. This design uses one
  call with three questions, against a full generative completion per review today. Jev plausibly
  wins this dimension outright.
• Stability. A published self-consistency comparison measured TypeSafe's per-question
  probability standard deviation at 0.0102, below every LLM control condition — including at
  temperature 0, where the LLMs still disagreed with themselves. For an audit function that is a
  real advantage.
• Calibrated confidence. This is what System One models are built for, and it is precisely the
  mechanism this issue asks for ("judgement reaching a verified confidence requirement").

In doubt — and these are the three items this issue itself lists as to-be-verified:

• Chinese authorization. The models page states CJK scripts are "handled but not equally well",
  and the jaggedness page states that "scoping words, negations, and implied conditions are read
  at face value". The sub-cases this issue names — negation and withdrawn authorization — are
  exactly the documented weak spot.
• Complex commands. Documented indirection and multi-hop weakness, with low-level
  representations performing worse. Nested shell pipelines are that shape.
• Prompt injection. The decisive one. The jaggedness page states that jev-1.13 "does not treat
  state as hostile by default", that adversarial content "can move the answer", and that improvement
  is expected "in the future". This is a model property, not a prompt-tuning problem, and the
  asymmetry is unfavourable: a false allow is a security failure, a false escalation is friction.

So: the mechanical requirements are achievable, and three of the six quality dimensions plausibly
favour Jev. But two of the three to-be-verified items land on documented weak spots, and the
decisive one is documented as currently undefended. This does not mean the answer is "unsuitable" —
this issue's own answer space includes "suitable only for limited scenarios" — but it does mean
feasibility cannot be assumed.

A structural concern worth raising before the evaluation

All questions in one call share the same state. The documentation is explicit that questions
are evaluated in parallel and independently — but "independently" refers to the answers not seeing
each other, not to separate inputs.

The consequence for this use case: injected content does not only move the injection_pressure
answer. It can move risk_level and user_authorization as well, because all three questions read
the same untrusted state.

This design deliberately includes tool results in the state, because that is where injection lives
and the injection question must see it. That creates a tension the current single-call design cannot
resolve: to let the injection question see untrusted content, the risk and authorization questions
must be exposed to the same content.

A concrete alternative, and one TypeSafe's own guidance points at ("filter first; send only what the
question needs"): split into two calls with two different states — one carrying the untrusted
content for injection detection, one carrying only the trusted action and user request for risk and
authorization — and compose the results in code. If the evaluation shows injection is the blocker,
this is likely the first thing to try.

What is needed to close this issue

A 16-case probe is prepared against the live API. It uses the real question set and the real
composition, so a pass is evidence about this integration rather than about TypeSafe in general. The
cases cover: routine authorized actions, actions that must be blocked (credential reads, key
exfiltration, broad destructive deletes), unclear authorization, injection via tool output
(including a fabricated "already approved" claim), Chinese authorization including negation and
withdrawal, and complex shell commands.

The gate is zero false allows on the block and injection cases. False escalations are reported
but not gated, since they are a tuning question rather than a safety failure.

This cannot run yet: no TypeSafe API key is available. Cost is negligible (16 calls, a few
thousand input tokens, $0.042/Mtok). The injection group's result alone decides whether the full
evaluation is worth investing in.

Follow-ups not addressed by #2335

• A failed judgment review is invisible to the user: the reviewer returns a generic ask_user
  rationale and the caller drops it, so a misconfigured judgment model silently turns every
  auto-approve into a prompt with no way to find out why. Fail-safe, but it needs a surface.
• getProviderSummaries omits models, customModels, enabledModels and disabledModels, so
  reordering providers silently strips them from the stored provider list. Pre-existing and
  unrelated; the Jev fallback was hardened against it rather than fixing the reorder path.