Skip to content

feat(agent-interface): unify exact candidate outcomes and receipts#46

Merged
drewstone merged 4 commits into
mainfrom
feat/universal-candidate-outcome
Jul 13, 2026
Merged

feat(agent-interface): unify exact candidate outcomes and receipts#46
drewstone merged 4 commits into
mainfrom
feat/universal-candidate-outcome

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Problem

Candidate execution had parallel task-output shapes and duplicated receipt/usage records, making exact workspace and non-workspace outcomes hard to share across products.

Solution

  • define one exact task outcome contract for workspace or output results
  • collapse execution receipt and usage shapes
  • remove obsolete duplicate contracts and parsers

Verification

  • 105 agent-interface tests passed
  • full workspace typecheck, test, and build passed
  • branch merges cleanly with current main

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 1146e073

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-13T07:26:24Z

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Value Audit — sound

Verdict sound
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 464.3s (2 bridge agents)
Total 464.3s

💰 Value — sound

Unifies the pre-release candidate receipt around one discriminated task-outcome union (workspace or bounded media output) and collapses the V1/V2 receipt and model-settlement duplication into a single canonical shape; built in the grain of the existing codebase with all 105 tests passing.

  • What it does: Three coupled changes to the @tangle-network/agent-interface (v0.24.0, explicitly pre-release) candidate evidence contract. (1) Introduces AgentCandidateTaskOutcomeSpec as a kind-discriminated union (workspace | output) and threads it through the execution plan and the task-outcome material, replacing the workspace-only task.repository field and adding a new bounded-output variant where a candid
  • Goals it achieves: (a) Let the same receipt/grading pipeline carry either code-workspace results OR bounded non-code outputs (data extraction, JSON, etc.) through one discriminated contract, so products stop forking parallel outcome shapes. (b) Kill the V1/V2 dual-parser machinery before the contract stabilizes — the version split existed only because V2 added fields on top of V1 instead of folding back, and carryin
  • Assessment: Coherent and in-grain. The new kind-tagged outcome union mirrors how the rest of the candidate contract already models choices (AgentCandidateCode, AgentCandidateMemoryReceipt, AgentCandidateTermination are all kind-discriminated unions), so this extends the existing pattern rather than introducing a new one. The refactor is genuinely subtractive: net -270 lines across 10 files, with the V1/V2 r
  • Better / existing approach: none — this is the right approach. I checked: (1) agentCandidateMediaTypeSchema is defined once in agent-candidate-schema-common.ts:43 and reused, not duplicated. (2) The discriminated-union shape matches the existing AgentCandidateCode pattern (agent-candidate.ts:200-203), so this folds into the codebase's grain rather than fighting it. (3) The pre-PR state on main was strictly worse — parallel V
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

Clean unification of the candidate-outcome contract: extends the plan/result shapes with a bounded typed-output branch and collapses the redundant V1/V2 receipt and settlement schemas into one canonical V1 each — built in the grain of a pre-release contract package ahead of its first external consum

  • Integration: Reachable via the public export chain index.ts:809 → agent-candidate-schema.ts:171,173 → agent-candidate-outcome-schema.ts / agent-candidate-receipt-schema.ts. README:80 already tells producers to attach AgentCandidateRunReceipt to benchmark runs. No sibling workspace package consumes any candidate schema yet, but the candidate contract is brand-new (#41/#43/#45 landed days ago) and the package is
  • Fit with existing patterns: Idiomatic. Every neighbor contract uses z.discriminatedUnion + .strict() + .superRefine + isCanonicalJsonValue (e.g. agent-candidate-outcome-schema.ts:214-300, agent-candidate-execution-plan-schema.ts:84-101). The new output branch reuses agentCandidateMediaTypeSchema and agentCandidateArtifactRefSchema rather than forking primitives. The V1/V2 collapse removes a real duplication (prior receipt ca
  • Real-world viability: 105 tests pass and tsc --noEmit is clean. New refinements are correct: maxBytes ≤ 64 MiB and positive (agent-candidate-execution-plan-schema.ts:232), output artifact byteLength > 0 (agent-candidate-outcome-schema.ts:284-293), and the empty-task-workspace check is correctly gated on outcome.kind === "workspace" so bounded-output tasks aren't falsely rejected (agent-candidate-execution-plan-schema.t
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260713T073617Z

@tangletools

Copy link
Copy Markdown

✅ No Blockers — 1146e073

Review health 100/100 · Reviewer score 83/100 · Confidence 75/100 · 4 findings (4 low)

glm: Correctness 83 · Security 83 · Testing 83 · Architecture 83

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 3/3 planned shots over 10 changed files. Global verifier still owns final merge decision.

🟡 LOW Changelog phrasing is opaque to consumers — .changeset/exact-task-outputs.md

The line 'Break the pre-release candidate evidence contract to bind and receipt either exact workspace changes or bounded media outputs through one result union.' is dense and uses insider vocabulary ('evidence contract', 'bind and receipt', 'result union') that will appear verbatim in the public CHANGELOG. Consider a plainer second sentence aimed at consumers, e.g. 'Result payloads now carry either workspace diffs or media outputs as a single discriminated union; update pattern matches accordingly.' Cosmetic only; does not block merge.

🟡 LOW Output artifact byteLength not cross-checked against plan maxBytes — packages/agent-interface/src/agent-candidate-outcome-schema.ts

The outcome material's output-kind refine only rejects byteLength === 0; there is no upper-bound enforcement against plan.task.outcome.maxBytes (capped at 64 MiB in agent-candidate-execution-plan-schema.ts:232). The run receipt carries only executionPlanDigest, not the plan material, so the validator literally cannot see maxBytes here — the cross-check would have to live in a higher layer that joins both receipts. Impact: a buggy or hostile producer could capture a >64 MiB artifact while the plan promised less; the schema alone will not catch it. Acceptable given the digest-binding model (grader is the ultimate verifier of bytes), but worth a comment or a future cross-receipt verifier. Not blocking.

🟡 LOW mediaType regex is stricter than RFC 6838 (intentional canonical subset) — packages/agent-interface/src/agent-candidate-schema-common.ts

Pattern /^[a-z0-9][a-z0-9!#$&^.+-]{0,126}/[a-z0-9][a-z0-9!#$&^.+-]{0,126}$/ enforces lowercase only and excludes '*' (allowed by RFC 6838 in some vendor trees). Canonical-form restriction is consistent with the rest of the package's RFC 8785 canonical-bytes philosophy, and the test confirms 'Application/JSON' is rejected. Common media types (application/json, application/vnd.api+json, text/plain, image/png) all match. Not a defect — flagging only so reviewers know the canonical-subset decision is deliberate.

🟡 LOW Output-kind task outcome material omits mediaType, forcing plan join — packages/agent-interface/src/agent-candidate.ts

AgentCandidateTaskOutcomeMaterialV1.outcome for kind="output" carries only { artifact: AgentCandidateArtifactRef }. The declared mediaType lives exclusively on AgentCandidateTaskOutcomeSpec in the execution plan. A consumer reading the outcome material in isolation cannot tell whether the captured bytes are application/json vs text/plain without resolving executionPlanDigest back to the plan. The doc comment on mediaType ('the bound grader verifies the bytes match it') makes this an intentional design choice (grader is the verifier), and the artifact schema carries no mediaType field either. Defensible, but a usability wrinkle for receipt-only consumers.


tangletools · 2026-07-13T07:47:22Z · trace

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — 4 non-blocking findings — 1146e073

Full multi-shot audit completed 3/3 planned shots over 10 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary


tangletools · 2026-07-13T07:47:22Z · immutable trace

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — fdbe7d34

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-13T15:15:05Z

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 451f6518

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-13T15:44:39Z

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Value Audit — sound

Verdict sound
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 216.5s (2 bridge agents)
Total 216.5s

💰 Value — sound

Collapses parallel V1/V2 candidate contracts into one canonical schema set, adds a workspace|output discriminated task outcome, folds the full promotion pipeline (propose→review→execute) into agent-interface, and replaces duplicated float-dollar usage tracking with one integer-nanos per-call settlem

  • What it does: (1) Removes the parallel V1/V2 split across bundle, run/materialization receipts, profile-plan, execution-plan, workspace-manifest, model-settlement, task-outcome and benchmark-result materials — every *V1/*V2 type alias and *V1Schema/*V2Schema zod parser is collapsed to a single canonical form, keeping the richer V2 shape (e.g. generationId on settlement calls) as the only form. (2) Int
  • Goals it achieves: Read from the change itself: (a) one canonical task-outcome contract so workspace and non-workspace (bounded-media) tasks share the same execution-plan, receipt, and benchmark-result shapes; (b) kill the duplicated/transitional usage tracking — the old usage + modelUsage pair and the V1/V2 settlement split are gone, replaced by one router-authored ledger with full per-call detail and exact int
  • Assessment: Good change on its merits, in the grain of the codebase. agent-interface already owned the bundle/receipt contracts; extending it to own the promotion contract and collapse the V1/V2 split is the natural end state of the transitional versioning added in #33 (executable outcome receipts) and #43 (benchmark graders). The discriminated workspace|output outcome is the right factorization — previousl
  • Better / existing approach: none — this is the right approach. Searched: git grep for AgentImprovementProposal|AgentImprovementReview|CandidateExecutionEvidence|agent-candidate-promotion-schema (only matches are inside this package — nothing else in the repo defines a promotion contract, so no duplication); git grep for every removed *V1/*V2 symbol (zero hits — no internal consumer breaks); checked ls packages/ (
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

Clean consolidation: collapses an internal V1/V2 receipt split, adds a unified workspace-vs-output task outcome discriminated contract, and lands the proposal→review→execution-evidence promotion chain — all in-grain with the existing schema pattern and exported from a contracts package whose callers

  • Integration: All new schemas are exported via packages/agent-interface/src/index.ts:810 (export * from "./agent-candidate-promotion-schema.js") and via the existing re-exports in agent-candidate-schema.ts:171-173. The package is a published contracts library (@tangle-network/agent-interface@0.24.0); no package inside this monorepo consumes the candidate surface today (grepped: agent-core, agent-provider-{tan
  • Fit with existing patterns: Follows the established pattern exactly: z.object({...}).strict().superRefine(...) satisfies z.ZodType; MutuallyAssignable<A,B> compile-time type/schema parity assertions; discriminated unions on kind; reuse of shared helpers (agentCandidateMediaTypeSchema, sha256Utf8, isWellFormedUnicode, isSafeRelativePath, isCanonicalJsonValue) from agent-candidate-schema-common.ts. Collapses a V1/V2 split
  • Real-world viability: Discriminated outcome contract is robust: workspace outcomes require task.repository provenance (agent-candidate-execution-plan-schema.ts:392-401) and enforce matching base/result repo identity and Git-object-format consistency (agent-candidate-outcome-schema.ts:251-290); output outcomes reject empty artifacts and artifacts exceeding the signed maxBytes (agent-candidate-outcome-schema.ts:291-310).
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260713T161202Z

@tangletools

Copy link
Copy Markdown

✅ No Blockers — 451f6518

Review health 100/100 · Reviewer score 77/100 · Confidence 85/100 · 6 findings (6 low)

glm: Correctness 77 · Security 77 · Testing 77 · Architecture 77

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 5/5 planned shots over 18 changed files. Global verifier still owns final merge decision.

🟡 LOW Changeset summary uses dense insider phrasing — .changeset/exact-task-outputs.md

The body line 'Those records bind the exact baseline and candidate bundle, preserve every held-out objective, carry full knowledge promotion identity, and include the materialization receipt plus native profile files used for execution' stacks ~5 domain terms (baseline/candidate bundle, held-out objective, knowledge promotion identity, materialization receipt, native profile files) into one sentence. Future changelog readers will struggle to map this to user-visible behavior. Consider splitting into 2-3 shorter sentences focused on what changed for consumers. Non-blocking; purely editorial.

🟡 LOW Workspace manifest mode bits relaxed from strict whitelist to full 0o000-0o777 range — packages/agent-interface/src/agent-candidate-artifact-schema.ts

The file mode constraint changed from z.union([z.literal(0o644), z.literal(0o755)]) to z.number().int().min(0).max(0o777). This allows world-writable (0o777), zero-permission (0o000), and setuid-group combinations. The signed manifest provides post-execution integrity verification, so tampering is detectable, but the previous whitelist was defense-in-depth against accidental or adversarial permission misuse inside the container. The type comment says 'Exact regular-file permission bits, excluding file-type and special bits' which is technically correct for 0-0o777, but the relaxation means the schema no longer enforces safe-default file permissions. Impact: low — the workspace is isolated and post-execution snapshots detect modifications. Fix: consider narrowing to common safe modes (e

🟡 LOW Profile activation schema does not explicitly reject duplicate file paths — packages/agent-interface/src/agent-candidate-execution-plan-schema.ts

The activation schema checks activation.files[index] against planned[index] by position, but does not call addDuplicateIssues on activation.files.map(f => f.path). If two activation files share a path, the error would surface as 'path must match the canonical plan' (positional mismatch) rather than 'duplicate path'. The planned files are deduplicated and sorted by the profile plan schema, so duplicates in the activation would be caught indirectly, but the error message is misleading. Impact: low — validation catches the issue, just with a confusing message. Fix: add addDuplicateIssues(activation.files.map(f => f.path), ['files'], ctx) before the index loop.

🟡 LOW AgentCandidateKnowledge schema does not bind candidate.candidateHash to snapshot.digest — packages/agent-interface/src/agent-candidate-lineage-schema.ts

The restructured knowledge contract carries candidate.candidateHash (a Sha256Digest) alongside snapshot.digest (also a Sha256Digest). The test fixture sets both to candidateSha("7"), indicating the intended binding. However, the schema has no superRefine checking candidate.candidateHash === snapshot.digest. A malformed or adversarial knowledge block could claim a candidateHash that does not match the snapshot digest, breaking the integrity chain at the schema level. The runtime verifier presumably recomputes and verifies, but other schemas in this codebase (e.g., execution plan binding profile plan digest) enforce cross-field digest consistency at the schema level. Impact: low — runtime verifier catches mismatches. Fix: add `.superRefine((k, ctx) => { if (k.candidate.candidateHash

🟡 LOW Measured comparison enforces power.n === overall.n but not per-objective n consistency — packages/agent-interface/src/agent-candidate-promotion-schema.ts

The superRefine checks comparison.power.n !== comparison.overall.n but does not verify that each measured objective's n equals comparison.overall.n. In a paired held-out comparison, all measured objectives should use the same paired sample. An objective with availability: "measured" and a different n (e.g., n=10 when overall.n=12) would pass schema validation but be methodologically invalid for a paired bootstrap. Impact: low — eval producers are expected to be well-behaved, but the schema is the natural enforcement point. Fix: after the objective loop, check if (objective.availability === "measured" && objective.n !== comparison.overall.n) ctx.addIssue(...).

🟡 LOW candidateExecutionEvidenceSchema does not verify receipt.bundleDigest against proposal's bundle — packages/agent-interface/src/agent-candidate-promotion-schema.ts

The evidence schema checks receipt.bundleDigest === materialization.bundleDigest (in-document binding) but does not carry the proposal document, so it cannot verify that the receipt's bundle matches the proposal's candidateBundle.digest. The evidence carries only proposalDigest (a hash). This is a cross-document binding that must be enforced by the consuming service. The evidence schema DOES verify in-document consistency thoroughly (7 invariant checks). Impact: low — by-design cross-document pattern. Fix: document the required service-level check or embed the proposal digest chain explicitly.


tangletools · 2026-07-13T16:22:19Z · trace

@tangletools tangletools left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved — 6 non-blocking findings — 451f6518

Full multi-shot audit completed 5/5 planned shots over 18 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary


tangletools · 2026-07-13T16:22:19Z · immutable trace

@drewstone drewstone merged commit 7e34b8c into main Jul 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants