feat(llm): let cloud providers pass vendor-specific body fields - #149
Merged
Conversation
`buildOpenAiChatBody` assembled a fixed set of OpenAI schema fields, so a deployment could not reach vendor extensions that live outside that schema. Alibaba Model Studio gates reasoning through `chat_template_kwargs.enable_thinking`, and there was no way to send it short of a code change per vendor. Add an optional `extraBody` object to `openai-compatible` and `qwen-openai-compatible` provider entries. It is merged into the chat completion body, then `model`, `messages`, `stream` and `tools` are re-applied on top, so a config entry can never detach the request from the resolved model, rewrite the prompt, flip streaming, or smuggle in a tool contract the caller did not set. Scope notes: - Local (`llama-server`) requests are untouched. Tool calling there runs through GBNF, and `sampling-filter` already strips `grammar` before any cloud call, so the two paths stay separate. - When `extraBody` is absent the body is byte-identical to before. - `openrouter`, `aimlapi` and `gemini` are left alone; they can opt in later if a vendor knob calls for it. Motivation: measuring Qwen thinking modes (full / capped / off) through the agent instead of a side script, so every arm runs under identical conditions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
buildOpenAiChatBodyassembles a fixed set of OpenAI schema fields, so a deployment cannot reach vendor extensions that live outside that schema.Concretely: Alibaba Model Studio gates reasoning through
chat_template_kwargs.enable_thinking. There is currently no way to send it short of adding a bespoke field per vendor.Change
Optional
extraBodyobject onopenai-compatibleandqwen-openai-compatibleprovider entries, merged into the chat completion body.{ "id": "model-studio", "kind": "qwen-openai-compatible", "baseUrl": "https://dashscope-intl.aliyuncs.com/compatible-mode", "defaultChatModel": "qwen3.8-27b", "extraBody": { "chat_template_kwargs": { "enable_thinking": false } } }model,messages,streamandtoolsare re-applied after the merge. A config entry cannot detach the request from the resolved model, rewrite the prompt, flip streaming, or smuggle in a tool contract the caller never set (the last case is covered by a test).Scope notes
llama-server) requests are untouched. Tool calling there runs through GBNF, andsampling-filteralready stripsgrammarbefore any cloud call, so the two paths never meet.extraBodyabsent the body is byte-identical to before, asserted by a test.openrouter,aimlapiandgeminiare deliberately left alone. They can opt in later if a vendor knob calls for it.extraBodyrather than passing it through.Testing
origin/mainat the time of writing (TUI / sidecar / fs-glob). No regressions introduced.Motivation
Measuring Qwen thinking modes (full / capped / off) through the agent rather than a side script, so every arm of the comparison runs under identical conditions.