Skip to content

feat(llm): let cloud providers pass vendor-specific body fields - #149

Merged
sosidudku1 merged 1 commit into
mainfrom
feat/provider-extra-body
Aug 18, 2026
Merged

feat(llm): let cloud providers pass vendor-specific body fields#149
sosidudku1 merged 1 commit into
mainfrom
feat/provider-extra-body

Conversation

@sosidudku1

Copy link
Copy Markdown
Collaborator

Problem

buildOpenAiChatBody assembles a fixed set of OpenAI schema fields, so a deployment cannot reach vendor extensions that live outside that schema.

Concretely: Alibaba Model Studio gates reasoning through chat_template_kwargs.enable_thinking. There is currently no way to send it short of adding a bespoke field per vendor.

Change

Optional extraBody object on openai-compatible and qwen-openai-compatible provider entries, merged into the chat completion body.

{
  "id": "model-studio",
  "kind": "qwen-openai-compatible",
  "baseUrl": "https://dashscope-intl.aliyuncs.com/compatible-mode",
  "defaultChatModel": "qwen3.8-27b",
  "extraBody": { "chat_template_kwargs": { "enable_thinking": false } }
}

model, messages, stream and tools are re-applied after the merge. A config entry cannot detach the request from the resolved model, rewrite the prompt, flip streaming, or smuggle in a tool contract the caller never set (the last case is covered by a test).

Scope notes

  • Local (llama-server) requests are untouched. Tool calling there runs through GBNF, and sampling-filter already strips grammar before any cloud call, so the two paths never meet.
  • With extraBody absent the body is byte-identical to before, asserted by a test.
  • openrouter, aimlapi and gemini are deliberately left alone. They can opt in later if a vendor knob calls for it.
  • Config parsing rejects a non-object extraBody rather than passing it through.

Testing

  • 6 new unit tests (4 body builder, 2 config parser).
  • Full suite: 6 failing files, identical set to origin/main at the time of writing (TUI / sidecar / fs-glob). No regressions introduced.

Motivation

Measuring Qwen thinking modes (full / capped / off) through the agent rather than a side script, so every arm of the comparison runs under identical conditions.

`buildOpenAiChatBody` assembled a fixed set of OpenAI schema fields, so
a deployment could not reach vendor extensions that live outside that
schema. Alibaba Model Studio gates reasoning through
`chat_template_kwargs.enable_thinking`, and there was no way to send it
short of a code change per vendor.

Add an optional `extraBody` object to `openai-compatible` and
`qwen-openai-compatible` provider entries. It is merged into the chat
completion body, then `model`, `messages`, `stream` and `tools` are
re-applied on top, so a config entry can never detach the request from
the resolved model, rewrite the prompt, flip streaming, or smuggle in a
tool contract the caller did not set.

Scope notes:

- Local (`llama-server`) requests are untouched. Tool calling there runs
  through GBNF, and `sampling-filter` already strips `grammar` before any
  cloud call, so the two paths stay separate.
- When `extraBody` is absent the body is byte-identical to before.
- `openrouter`, `aimlapi` and `gemini` are left alone; they can opt in
  later if a vendor knob calls for it.

Motivation: measuring Qwen thinking modes (full / capped / off) through
the agent instead of a side script, so every arm runs under identical
conditions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sosidudku1
sosidudku1 merged commit 90ad218 into main Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant