fix(foreman): tell a spin from a real max-turns burn - #1814
Conversation
A loop can burn all 160 turns in 65s (~0.4s/turn, faster than any model can generate) and still report the bare MaxTurnsExhausted that a genuine 1662s exhaustion reports. Track per-turn productivity (completions, tool calls, time in turn), classify a fast exhaustion as the new LoopSpinning reason, and carry the counters into the audit record so a spin is diagnosable from the record alone. Signed-off-by: Jory Irving <jory@jory.dev>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Defilan
left a comment
There was a problem hiding this comment.
Approving, the spin trace holds up end to end. I re-derived the threshold arithmetic (1s times Turns is Turns seconds, not Turns nanoseconds) and ran the real 160-turn/65s and 1662s cases through mapLoopError; they classify the way the test says. Both new tests bite: I removed the two counter increments in loop.go and the guard condition in executor_native.go, and the respective tests failed with the counters at zero and the reason reverting to MaxTurnsExhausted. The guard's edges hold too; I probed 9 turns at a zero duration, a zero-value LoopResult, and both sides of the 1s/turn boundary, and only the sub-threshold side classifies. The audit wiring is consistent: incompleteResult writes turnsWithCompletion / turnsWithToolCall and resultPayload reads the same keys, and a pre-fix result JSON stays zero rather than fabricated. CRDs sync and CI is green.
One thing before merge: the body says Fixes #1628, but the issue's first ask is that a turn which neither completes nor calls a tool must not consume the budget silently, either by excluding it from the count or aborting with a distinct reason. This reports the ratio after the fact and does not stop the burn, so merging would close an issue that still has an open ask. Refs #1628 plus a follow-up, or fold that in.
Thanks for the boundary coverage, it made this quick to check.
…25 ➔ 0.9.27) (#2111) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/defilantech/charts/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.25` → `0.9.27` | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information. --- ### Release Notes <details> <summary>defilantech/LLMKube (ghcr.io/defilantech/charts/llmkube)</summary> ### [`v0.9.27`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0927-2026-09-14) [Compare Source](defilantech/LLMKube@v0.9.26...v0.9.27) ##### Bug Fixes - **deps:** bump google.golang.org/grpc to the CVE-2026-84445 fix ([#​1823](defilantech/LLMKube#1823)) ([cff58b5](defilantech/LLMKube@cff58b5)) ### [`v0.9.26`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0926-2026-09-14) [Compare Source](defilantech/LLMKube@v0.9.25...v0.9.26) ##### Features - **controller:** metric-gated idle probe for the generic runtime ([#​1802](defilantech/LLMKube#1802)) ([750be58](defilantech/LLMKube@750be58)) - **controller:** ModelPool reclaim policy returns the slot to spec.default after idle ([#​1796](defilantech/LLMKube#1796)) ([9fd13cd](defilantech/LLMKube@9fd13cd)) - **foreman:** expose node label on FleetNode metrics for fleet joins ([#​1785](defilantech/LLMKube#1785)) ([5db0f7c](defilantech/LLMKube@5db0f7c)) - **foreman:** serve Agents from Anthropic-native endpoints ([#​1811](defilantech/LLMKube#1811)) ([a2e2b2e](defilantech/LLMKube@a2e2b2e)) - **inferenceservice:** add resources.memoryLimit to decouple ceiling from reservation ([#​1764](defilantech/LLMKube#1764)) ([53e5090](defilantech/LLMKube@53e5090)) - **model:** resume interrupted single-file downloads via a content-keyed partial ([#​1765](defilantech/LLMKube#1765)) ([#​1807](defilantech/LLMKube#1807)) ([e3384f4](defilantech/LLMKube@e3384f4)) - **router:** mount /v1/embeddings and /v1/rerank on the proxy ([#​1819](defilantech/LLMKube#1819)) ([3af89e9](defilantech/LLMKube@3af89e9)) - **router:** opt-in endpoint-level backends for per-request balancing ([#​1820](defilantech/LLMKube#1820)) ([d9349e2](defilantech/LLMKube@d9349e2)) - **router:** poolActivation IfIdle serves the warm member when the incumbent is busy ([#​1787](defilantech/LLMKube#1787)) ([d211a53](defilantech/LLMKube@d211a53)) - **samples:** three-Spark ring DeepSeek-V4.1-Flash EXL3 TP3 samples and the multi-node guide section ([#​1816](defilantech/LLMKube#1816)) ([6cb8389](defilantech/LLMKube@6cb8389)) ##### Bug Fixes - **agent:** resolve hf:// model sources on the metal path ([#​1759](defilantech/LLMKube#1759)) ([#​1789](defilantech/LLMKube#1789)) ([0d4879c](defilantech/LLMKube@0d4879c)) - **controller:** hold a multiNode group while a member is staging ([#​1757](defilantech/LLMKube#1757)) ([#​1806](defilantech/LLMKube#1806)) ([116ba95](defilantech/LLMKube@116ba95)) - **controller:** probe multiNode worker rendezvous, fix readyMembers ([#​1781](defilantech/LLMKube#1781)) ([#​1782](defilantech/LLMKube#1782)) ([97acb8e](defilantech/LLMKube@97acb8e)) - **controller:** re-resolve a Model when spec.source changes ([#​1767](defilantech/LLMKube#1767)) ([#​1805](defilantech/LLMKube#1805)) ([9149419](defilantech/LLMKube@9149419)) - **foreman-chart:** name the FleetNode in the stale-heartbeat alert ([#​1779](defilantech/LLMKube#1779)) ([#​1783](defilantech/LLMKube#1783)) ([d4138d2](defilantech/LLMKube@d4138d2)) - **foreman:** anchor coder rail diffs to the resolved upstream base SHA ([#​1803](defilantech/LLMKube#1803)) ([08ce692](defilantech/LLMKube@08ce692)) - **foreman:** clear a FleetNode reservation whose AgenticTask was deleted ([#​1791](defilantech/LLMKube#1791)) ([#​1804](defilantech/LLMKube#1804)) ([0cc53c4](defilantech/LLMKube@0cc53c4)) - **foreman:** drain tasks before agent exit ([#​1799](defilantech/LLMKube#1799)) ([9b22e73](defilantech/LLMKube@9b22e73)) - **foreman:** omit issue references from the reviewer prompt when the payload carries none ([#​1761](defilantech/LLMKube#1761)) ([#​1808](defilantech/LLMKube#1808)) ([36e0e0b](defilantech/LLMKube@36e0e0b)) - **foreman:** reap orphaned in-cluster FleetNodes when agent pods disappear ([#​1778](defilantech/LLMKube#1778)) ([#​1792](defilantech/LLMKube#1792)) ([9fbee34](defilantech/LLMKube@9fbee34)) - **foreman:** reject unverified reviewer GO ([#​1801](defilantech/LLMKube#1801)) ([27910ed](defilantech/LLMKube@27910ed)) - **foreman:** tell a spin from a real max-turns burn ([#​1814](defilantech/LLMKube#1814)) ([3e93fe4](defilantech/LLMKube@3e93fe4)) - **foreman:** use the coder's PR description when the PR opens on review GO ([#​1768](defilantech/LLMKube#1768)) ([#​1776](defilantech/LLMKube#1776)) ([bff8344](defilantech/LLMKube@bff8344)) - **grafana:** separate decode throughput axis ([#​1815](defilantech/LLMKube#1815)) ([bf77989](defilantech/LLMKube@bf77989)) - **model:** re-resolve a Model whose spec changed when no cache key exists ([#​1818](defilantech/LLMKube#1818)) ([81be593](defilantech/LLMKube@81be593)) - **rollout:** do not defer a scale-from-zero rollout as PodsBusy ([#​1794](defilantech/LLMKube#1794)) ([062332f](defilantech/LLMKube@062332f)) - **router:** return 503 + Retry-After when every IfIdle backend skips busy ([#​1797](defilantech/LLMKube#1797)) ([396177d](defilantech/LLMKube@396177d)) ##### Documentation - **labs:** reference build for GLM-5.3-Flash EXL3 on two DGX Sparks ([#​1784](defilantech/LLMKube#1784)) ([cbdbb25](defilantech/LLMKube@cbdbb25)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Paris) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC44My4xIiwidXBkYXRlZEluVmVyIjoiNDQuODMuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2111
What
Give the foreman loop turn-accounting, and give a max-turns exhaustion that ran faster than a model physically can its own failure reason:
LoopSpinning. Theforeman.audit.v1record now carriesturnsWithCompletion/turnsWithToolCallalongsideturnCount.Why
Refs #1628. Two issue-fix tasks burned all 160 turns in 105s and 65s — 0.4–0.66 s/turn, faster than any model generates — and terminated
INCOMPLETEwith the same bareMaxTurnsExhaustedthat a genuine 1106s/1662s exhaustion reports. The audit record showed a turn count but no productivity signal, so a spin looked exactly like a model that gave up, and the operator's only lever (raise the budget / tighten the prompt) was the wrong one.How
pkg/foreman/agent/loop.go):LoopResultgainsTurnsWithCompletion(turns whose chat request yielded a choice — transport errors and empty-choices aborts don't count),TurnsWithToolCall, andTurnDuration(cumulative wall-clock insiderunOneTurn). Each counter is incremented at a single point per turn, so the temperature-retry path can't double-count andTurnsWithCompletion <= Turnsholds by construction.pkg/foreman/agent/executor_native.go): inmapLoopError, a max-turns exhaustion withTurns >= 10and average turn duration strictly below 1s is classifiedLoopSpinning, with the observed rate in the summary; the [BUG] max-turns INCOMPLETE reports "model did not call submit_result" when it called it and was accepted #1713 rejected-submissions detail is kept when present. The guard lives in the executor, not the loop: the loop still returns theErrMaxTurnsExhaustedsentinel and knows nothing about CRD reasons.LoopSpinning(api/foreman/v1alpha1/agentictask_types.go); CRDs regenerated (config/crd/bases,charts/foreman/templates/crds). Distinct fix path: check the endpoint/shim, not the prompt/budget. Runbook updated.pkg/foreman/audit/record.go): the two new counters flow throughresultPayload.Extrainto the record; absent (pre-fix) result JSON leaves them zero andomitemptydrops them, so old runs are never rewritten with fabricated values.Deliberately out of scope: the no-tool-call / reasoning-only / truncated streak errors already have bounded streak budgets (their own circuit breakers), so only the
MaxTurnsExhaustedpath — the unbounded one — gets the spin classification.Checklist
TestLoop_TurnAccounting,TestMapLoopError_MaxTurns_SpinGuardincl. floor/threshold boundaries, audit record presence + absence cases)make testpasses locallymake lintpasses locally (alsomake lint-allacross GOOS)git commit -s) per DCOdocs/site/foreman/runbook-m3.md)Assisted-by: OpenCode (qwen3.8-flash-next coordinator; loop counter placement reviewed against an independent local Gemma review pass); verified by running
make test,make lint,make lint-all,make generate/manifests/chart CRD sync, and the new table-driven tests. Note: commit is DCO sign-off only — my machine's 1Password GPG signer was unavailable in the agent shell, so the commit is not GPG-signed.