Skip to content

fix: use tool_responses role for gemma4 models in LiteLLM integration#5655

Open
jfrometa88 wants to merge 3 commits intogoogle:mainfrom
jfrometa88:main
Open

fix: use tool_responses role for gemma4 models in LiteLLM integration#5655
jfrometa88 wants to merge 3 commits intogoogle:mainfrom
jfrometa88:main

Conversation

@jfrometa88
Copy link
Copy Markdown

Link to Issue or Description of Change

Closes: #5650

Problem

Gemma4 models served via Ollama do not recognise role: "tool" in the message history. When tool results were appended using this role, the model entered an infinite loop re-calling the tool instead of generating a final response.

Two separate code paths were affected:

  1. _content_to_message_param — builds tool result messages from function_response parts.
  2. _ensure_tool_results — heals incomplete histories by injecting placeholder messages when a tool call has no matching result. This function had both the trigger condition and the fallback block hardcoded to role: "tool", so it neither recognised incoming tool_responses messages as valid resolutions nor generated the correct role in the placeholder it injected.

Solution

Introduced a model-aware role selector in both affected functions. When the model name contains gemma4, the role is set to "tool_responses"; otherwise it falls back to the standard "tool".

In _content_to_message_param:
tool_role = "tool_responses" if "gemma4" in model.lower() else "tool"

In _ensure_tool_results, a single expected_tool_role variable is computed once before the loop and reused consistently in three places: the healing trigger condition, the elif branch that clears resolved call IDs, and the post-loop fallback block. This eliminates the previous dual-condition pattern where the two if blocks were not mutually exclusive and could both fire on the same message.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added tests/unittests/test_lite_llm_gemma_tool_role.py.

Covers:

  • Single tool response: verifies that when the model name contains gemma4, the generated message uses role: "tool_responses" instead of role: "tool".
  • Multiple tool responses: verifies the same behaviour when multiple function_response parts are present in a single content object.
  • Non-gemma4 models: verifies that role: "tool" is preserved for all other model names.

Full test suite result: 5709 passed, 2300 warnings.

Manual End-to-End (E2E) Tests:

Setup:

  • Ollama running locally with gemma4 (e2b and e4b variants tested), qwen2.5-coder:b and qwen2.5-coder:7b
  • Google ADK agent with a single tool: get_comprehensive_refugee_help
  • LiteLLM as the provider bridge

Steps:

  1. Send a vague message → agent asks for clarification (no tool call). ✓
  2. Send a message with explicit need and city → agent calls the tool once, receives tool_responses, generates final answer in the user's language. ✓
  3. Confirmed via LiteLLM debug logs that the outgoing message history contains only role: "tool_responses" for gemma4 models and "tool" for qwen models, without infinite loops.

Before the fix for gemma4 models: history contained both a phantom role: "tool" error message and the real role: "tool_responses" result, causing the model to process the error last and loop.
After the fix: history contains only the real role: "tool_responses" result.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[LiteLLM] Gemma models via Ollama enter infinite tool-calling loop due to wrong tool message role

1 participant