Skip to content

fix(foreman): tell a spin from a real max-turns burn - #1814

Merged
Defilan merged 1 commit into
defilantech:mainfrom
joryirving:fix/foreman-turn-accounting-1628
Sep 14, 2026
Merged

Defilan merged 1 commit into
defilantech:mainfrom
joryirving:fix/foreman-turn-accounting-1628

Conversation

@joryirving

@joryirving joryirving commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

What

Give the foreman loop turn-accounting, and give a max-turns exhaustion that ran faster than a model physically can its own failure reason: LoopSpinning. The foreman.audit.v1 record now carries turnsWithCompletion / turnsWithToolCall alongside turnCount.

Why

Refs #1628. Two issue-fix tasks burned all 160 turns in 105s and 65s — 0.4–0.66 s/turn, faster than any model generates — and terminated INCOMPLETE with the same bare MaxTurnsExhausted that a genuine 1106s/1662s exhaustion reports. The audit record showed a turn count but no productivity signal, so a spin looked exactly like a model that gave up, and the operator's only lever (raise the budget / tighten the prompt) was the wrong one.

How

  • Loop accounting (pkg/foreman/agent/loop.go): LoopResult gains TurnsWithCompletion (turns whose chat request yielded a choice — transport errors and empty-choices aborts don't count), TurnsWithToolCall, and TurnDuration (cumulative wall-clock inside runOneTurn). Each counter is incremented at a single point per turn, so the temperature-retry path can't double-count and TurnsWithCompletion <= Turns holds by construction.
  • Spin guard (pkg/foreman/agent/executor_native.go): in mapLoopError, a max-turns exhaustion with Turns >= 10 and average turn duration strictly below 1s is classified LoopSpinning, with the observed rate in the summary; the [BUG] max-turns INCOMPLETE reports "model did not call submit_result" when it called it and was accepted #1713 rejected-submissions detail is kept when present. The guard lives in the executor, not the loop: the loop still returns the ErrMaxTurnsExhausted sentinel and knows nothing about CRD reasons.
  • New API enum value LoopSpinning (api/foreman/v1alpha1/agentictask_types.go); CRDs regenerated (config/crd/bases, charts/foreman/templates/crds). Distinct fix path: check the endpoint/shim, not the prompt/budget. Runbook updated.
  • Thresholds: 10 turns floor / 1 s average cap, derived from the observed spins (0.41–0.66 s/turn) vs healthy runs (~1.7 s+/turn); the floor keeps tiny fast runs on the ordinary reason.
  • Audit (pkg/foreman/audit/record.go): the two new counters flow through resultPayload.Extra into the record; absent (pre-fix) result JSON leaves them zero and omitempty drops them, so old runs are never rewritten with fabricated values.

Deliberately out of scope: the no-tool-call / reasoning-only / truncated streak errors already have bounded streak budgets (their own circuit breakers), so only the MaxTurnsExhausted path — the unbounded one — gets the spin classification.

Checklist

  • Tests added/updated (TestLoop_TurnAccounting, TestMapLoopError_MaxTurns_SpinGuard incl. floor/threshold boundaries, audit record presence + absence cases)
  • make test passes locally
  • make lint passes locally (also make lint-all across GOOS)
  • Commit messages follow conventional commits
  • All commits are signed off (git commit -s) per DCO
  • AI assistance disclosed below, per CONTRIBUTING.md
  • Documentation updated (docs/site/foreman/runbook-m3.md)

Assisted-by: OpenCode (qwen3.8-flash-next coordinator; loop counter placement reviewed against an independent local Gemma review pass); verified by running make test, make lint, make lint-all, make generate/manifests/chart CRD sync, and the new table-driven tests. Note: commit is DCO sign-off only — my machine's 1Password GPG signer was unavailable in the agent shell, so the commit is not GPG-signed.

A loop can burn all 160 turns in 65s (~0.4s/turn, faster than any
model can generate) and still report the bare MaxTurnsExhausted that
a genuine 1662s exhaustion reports. Track per-turn productivity
(completions, tool calls, time in turn), classify a fast exhaustion
as the new LoopSpinning reason, and carry the counters into the
audit record so a spin is diagnosable from the record alone.

Signed-off-by: Jory Irving <jory@jory.dev>
@joryirving
joryirving requested a review from Defilan as a code owner September 13, 2026 12:11
@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Defilan Defilan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, the spin trace holds up end to end. I re-derived the threshold arithmetic (1s times Turns is Turns seconds, not Turns nanoseconds) and ran the real 160-turn/65s and 1662s cases through mapLoopError; they classify the way the test says. Both new tests bite: I removed the two counter increments in loop.go and the guard condition in executor_native.go, and the respective tests failed with the counters at zero and the reason reverting to MaxTurnsExhausted. The guard's edges hold too; I probed 9 turns at a zero duration, a zero-value LoopResult, and both sides of the 1s/turn boundary, and only the sub-threshold side classifies. The audit wiring is consistent: incompleteResult writes turnsWithCompletion / turnsWithToolCall and resultPayload reads the same keys, and a pre-fix result JSON stays zero rather than fabricated. CRDs sync and CI is green.

One thing before merge: the body says Fixes #1628, but the issue's first ask is that a turn which neither completes nor calls a tool must not consume the budget silently, either by excluding it from the count or aborting with a distinct reason. This reports the ratio after the fact and does not stop the burn, so merging would close an issue that still has an open ask. Refs #1628 plus a follow-up, or fold that in.

Thanks for the boundary coverage, it made this quick to check.

@Defilan
Defilan merged commit 3e93fe4 into defilantech:main Sep 14, 2026
25 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 14, 2026
eleboucher pushed a commit to eleboucher/homelab that referenced this pull request Sep 15, 2026
…25 ➔ 0.9.27) (#2111)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/defilantech/charts/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.25` → `0.9.27` |

---

> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information.

---

### Release Notes

<details>
<summary>defilantech/LLMKube (ghcr.io/defilantech/charts/llmkube)</summary>

### [`v0.9.27`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0927-2026-09-14)

[Compare Source](defilantech/LLMKube@v0.9.26...v0.9.27)

##### Bug Fixes

- **deps:** bump google.golang.org/grpc to the CVE-2026-84445 fix ([#&#8203;1823](defilantech/LLMKube#1823)) ([cff58b5](defilantech/LLMKube@cff58b5))

### [`v0.9.26`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0926-2026-09-14)

[Compare Source](defilantech/LLMKube@v0.9.25...v0.9.26)

##### Features

- **controller:** metric-gated idle probe for the generic runtime ([#&#8203;1802](defilantech/LLMKube#1802)) ([750be58](defilantech/LLMKube@750be58))
- **controller:** ModelPool reclaim policy returns the slot to spec.default after idle ([#&#8203;1796](defilantech/LLMKube#1796)) ([9fd13cd](defilantech/LLMKube@9fd13cd))
- **foreman:** expose node label on FleetNode metrics for fleet joins ([#&#8203;1785](defilantech/LLMKube#1785)) ([5db0f7c](defilantech/LLMKube@5db0f7c))
- **foreman:** serve Agents from Anthropic-native endpoints ([#&#8203;1811](defilantech/LLMKube#1811)) ([a2e2b2e](defilantech/LLMKube@a2e2b2e))
- **inferenceservice:** add resources.memoryLimit to decouple ceiling from reservation ([#&#8203;1764](defilantech/LLMKube#1764)) ([53e5090](defilantech/LLMKube@53e5090))
- **model:** resume interrupted single-file downloads via a content-keyed partial ([#&#8203;1765](defilantech/LLMKube#1765)) ([#&#8203;1807](defilantech/LLMKube#1807)) ([e3384f4](defilantech/LLMKube@e3384f4))
- **router:** mount /v1/embeddings and /v1/rerank on the proxy ([#&#8203;1819](defilantech/LLMKube#1819)) ([3af89e9](defilantech/LLMKube@3af89e9))
- **router:** opt-in endpoint-level backends for per-request balancing ([#&#8203;1820](defilantech/LLMKube#1820)) ([d9349e2](defilantech/LLMKube@d9349e2))
- **router:** poolActivation IfIdle serves the warm member when the incumbent is busy ([#&#8203;1787](defilantech/LLMKube#1787)) ([d211a53](defilantech/LLMKube@d211a53))
- **samples:** three-Spark ring DeepSeek-V4.1-Flash EXL3 TP3 samples and the multi-node guide section ([#&#8203;1816](defilantech/LLMKube#1816)) ([6cb8389](defilantech/LLMKube@6cb8389))

##### Bug Fixes

- **agent:** resolve hf:// model sources on the metal path ([#&#8203;1759](defilantech/LLMKube#1759)) ([#&#8203;1789](defilantech/LLMKube#1789)) ([0d4879c](defilantech/LLMKube@0d4879c))
- **controller:** hold a multiNode group while a member is staging ([#&#8203;1757](defilantech/LLMKube#1757)) ([#&#8203;1806](defilantech/LLMKube#1806)) ([116ba95](defilantech/LLMKube@116ba95))
- **controller:** probe multiNode worker rendezvous, fix readyMembers ([#&#8203;1781](defilantech/LLMKube#1781)) ([#&#8203;1782](defilantech/LLMKube#1782)) ([97acb8e](defilantech/LLMKube@97acb8e))
- **controller:** re-resolve a Model when spec.source changes ([#&#8203;1767](defilantech/LLMKube#1767)) ([#&#8203;1805](defilantech/LLMKube#1805)) ([9149419](defilantech/LLMKube@9149419))
- **foreman-chart:** name the FleetNode in the stale-heartbeat alert ([#&#8203;1779](defilantech/LLMKube#1779)) ([#&#8203;1783](defilantech/LLMKube#1783)) ([d4138d2](defilantech/LLMKube@d4138d2))
- **foreman:** anchor coder rail diffs to the resolved upstream base SHA ([#&#8203;1803](defilantech/LLMKube#1803)) ([08ce692](defilantech/LLMKube@08ce692))
- **foreman:** clear a FleetNode reservation whose AgenticTask was deleted ([#&#8203;1791](defilantech/LLMKube#1791)) ([#&#8203;1804](defilantech/LLMKube#1804)) ([0cc53c4](defilantech/LLMKube@0cc53c4))
- **foreman:** drain tasks before agent exit ([#&#8203;1799](defilantech/LLMKube#1799)) ([9b22e73](defilantech/LLMKube@9b22e73))
- **foreman:** omit issue references from the reviewer prompt when the payload carries none ([#&#8203;1761](defilantech/LLMKube#1761)) ([#&#8203;1808](defilantech/LLMKube#1808)) ([36e0e0b](defilantech/LLMKube@36e0e0b))
- **foreman:** reap orphaned in-cluster FleetNodes when agent pods disappear ([#&#8203;1778](defilantech/LLMKube#1778)) ([#&#8203;1792](defilantech/LLMKube#1792)) ([9fbee34](defilantech/LLMKube@9fbee34))
- **foreman:** reject unverified reviewer GO ([#&#8203;1801](defilantech/LLMKube#1801)) ([27910ed](defilantech/LLMKube@27910ed))
- **foreman:** tell a spin from a real max-turns burn ([#&#8203;1814](defilantech/LLMKube#1814)) ([3e93fe4](defilantech/LLMKube@3e93fe4))
- **foreman:** use the coder's PR description when the PR opens on review GO ([#&#8203;1768](defilantech/LLMKube#1768)) ([#&#8203;1776](defilantech/LLMKube#1776)) ([bff8344](defilantech/LLMKube@bff8344))
- **grafana:** separate decode throughput axis ([#&#8203;1815](defilantech/LLMKube#1815)) ([bf77989](defilantech/LLMKube@bf77989))
- **model:** re-resolve a Model whose spec changed when no cache key exists ([#&#8203;1818](defilantech/LLMKube#1818)) ([81be593](defilantech/LLMKube@81be593))
- **rollout:** do not defer a scale-from-zero rollout as PodsBusy ([#&#8203;1794](defilantech/LLMKube#1794)) ([062332f](defilantech/LLMKube@062332f))
- **router:** return 503 + Retry-After when every IfIdle backend skips busy ([#&#8203;1797](defilantech/LLMKube#1797)) ([396177d](defilantech/LLMKube@396177d))

##### Documentation

- **labs:** reference build for GLM-5.3-Flash EXL3 on two DGX Sparks ([#&#8203;1784](defilantech/LLMKube#1784)) ([cbdbb25](defilantech/LLMKube@cbdbb25))

</details>

---

### Configuration

📅 **Schedule**: (in timezone Europe/Paris)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC44My4xIiwidXBkYXRlZEluVmVyIjoiNDQuODMuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2111
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants