Skip to content

feat(xdev): the loop can write and verify — run_tests/write_file as real xdev turns - #33

Merged
linhdmn merged 1 commit into
mainfrom
feat/xdev-rpc
Sep 21, 2026
Merged

linhdmn merged 1 commit into
mainfrom
feat/xdev-rpc

Conversation

@linhdmn

@linhdmn linhdmn commented Sep 21, 2026

Copy link
Copy Markdown
Member

The question this answers

"What can this actually do in my repo?" I asked it to "fix the off-by-one in parseConfig and run the tests" and it rotated four tools and changed nothing — write_file returned written: false, run_tests returned passed: 0, output: "". Two of the four v1 tools were fiction, and the approval gate in front of them was theatre.

Now it writes and it verifies.

internal/xdev — the executor client

Speaks xdev's rpc JSONL protocol: a ready frame first (validated against protocol version 1, so a future v2 fails loudly instead of mis-parsing), then id-tagged prompts answered by a stream of events and one terminal response.

Four properties are load-bearing, and each has a test against a real child process rather than a mock:

Property Why it matters
the ready frame is a hard gate a child that dies without one fails Start instead of blocking a run forever
events are read until the response a client that waits for the response without draining events deadlocks behind a full pipe — xdev's own Serve comment names this
one turn at a time xdev refuses a second concurrent prompt, so serialising here turns a server-side error into a client-side wait
the step's context kills the child a turn that outlives its budget must not leave a half-read stream and a sandbox still mutating a workspace

The tools

run_tests and write_file each run as one xdev turn — which is PRD §4.1's contract in code: agentloop owns the loop and the bound, xdev owns the turn.

Three outcomes are made distinguishable, because "no sandbox" must never read as "the tests passed":

Situation Result
no executor configured Success, ran/written false, message says so
the turn failed Success=false with the child's reason — an observation the loop keeps
the turn ran Success, ran/written true, the turn's text in output

write_file with no path fails closed and says why.

The seam that was missing on the loop side

nextToolDefault passed only {step, goal}, so every write was rejected for a missing path before the sandbox ever saw it. It now gives each tool the arguments that make it a real call: the readers get the goal as query text, run_tests gets the workspace, write_file gets a target path. A real planner should name that file — this exists so the write path is exercisable end to end until it does, and the comment says so.

Wiring

AGENTLOOP_XDEV_BIN   default "xdev"
AGENTLOOP_XDEV_DIR   workspace turns run in (default: a fresh temp dir, because a
                     sandbox pointed at the server's cwd could edit agentloop itself)
AGENTLOOP_XDEV_OFF   disable

One child is shared across runs on purpose — a per-run child would pay startup on every step and lose the session between them — and it is still one turn at a time. A child that will not start is not fatal: the registry is built without a sandbox and the two tools report no executor. Same shape as LeanKG being down.

The M6 eval factory keeps no sandbox and no knowledge client: the deploy gate stays deterministic and offline, and still passes 4/4.

Verified live

Against a sandbox speaking the protocol:

step 2 run_tests  -> ran: true, output: "ran `go test ./...`: 12 passed, 0 failed"
step 3 write_file -> written: true, path: agentloop-<run>-step-3.txt

With AGENTLOOP_XDEV_OFF=1 the same run reports ran: false / written: false and names the missing executor. Against the real xdev binary the client reaches the ready frame and Start then fails on this machine's malformed ~/.xdev/agent/models.yml ("cannot unmarshal !!seq into string") — which is the correct shape: Start fails, the server logs a warning, and the run continues without a sandbox rather than dying.

What this does not yet make it

It still cannot choose its own steps: the planner is deterministic and nextToolDefault rotates. What changed is that the rotation now does something real — it writes and it runs tests. A model-driven planner is the next piece, and until it lands the loop is a working write-test harness with a bound and a gate, not an agent that finds the bug itself.

Checks

make check green: gofmt clean, vet clean, 14/14 packages (7 new tests in internal/xdev, 3 in internal/tools), lint 0 issues, PRD OK, selftest 12/12.

CI will show red on this PR — the org's Actions billing limit is still unraised (#31's caveat, recorded in USAGE §2). make check runs the identical five steps.

…eal xdev turns

Asked to "fix the off-by-one in parseConfig and run the tests", the loop
rotated four tools and changed nothing: `write_file` returned
`written: false` with a message saying `stub`, and `run_tests` returned
`passed: 0, output: ""`. Two of the four v1 tools were fiction, and the
approval gate in front of them was theatre.

## internal/xdev — the executor client

Speaks xdev's `rpc` JSONL protocol: `{"type":"ready"}` first (validated
against protocol version 1, so a future v2 fails loudly instead of
mis-parsing), then id-tagged prompts answered by a stream of events and one
terminal response.

Four properties are load-bearing, and each has a test against a real child
process rather than a mock:

  - **the ready frame is a hard gate** — a child that dies without one
    fails Start instead of blocking a run forever;
  - **events are read until the response** — a client that waits for the
    response without draining events deadlocks behind a full pipe, which
    xdev's own Serve comment names;
  - **one turn at a time** — xdev refuses a second concurrent prompt, so
    serialising here turns a server-side error into a client-side wait;
  - **the step's context kills the child** — a turn that outlives its
    budget must not leave a half-read stream and a sandbox still mutating
    a workspace.

## The tools

`run_tests` and `write_file` each run as one xdev turn. That is §4.1's
contract in code: agentloop owns the loop and the bound, xdev owns the turn
— the file mutation, the test run, the model call inside it.

Three outcomes are made distinguishable, because "no sandbox" must never
read as "the tests passed":

  | Situation | Result |
  |---|---|
  | no executor configured | Success, `ran`/`written` false, message says so |
  | the turn failed | Success=false with the child's reason — an observation the loop keeps |
  | the turn ran | Success, `ran`/`written` true, the turn's text in `output` |

`write_file` with no path fails closed and says why.

## The seam that was missing on the loop side

`nextToolDefault` passed only `{step, goal}`, so every write was rejected
for a missing path before the sandbox saw it. It now gives each tool the
arguments that make it a real call — the readers get the goal as query text,
`run_tests` gets the workspace, `write_file` gets a target. A real planner
should name the file; this exists so the write path is exercisable end to
end until it does, and the comment says so.

## Wiring

    AGENTLOOP_XDEV_BIN   default "xdev"
    AGENTLOOP_XDEV_DIR   workspace turns run in (default: a fresh temp dir,
                         because a sandbox pointed at the server's cwd could
                         edit agentloop itself)
    AGENTLOOP_XDEV_OFF   disable

One child is shared across runs on purpose (a per-run child would pay
startup per step and lose the session between them) and it is still one turn
at a time. A child that will not start is not fatal: the registry is built
without a sandbox and the two tools report no executor — the same shape as
LeanKG being down.

The M6 eval factory keeps **no** sandbox and no knowledge client: the deploy
gate stays deterministic and offline, and still passes 4/4.

## Verified live

Against a sandbox speaking the protocol:

    step 2 run_tests  -> ran: true,  output: "ran `go test ./...`: 12 passed, 0 failed"
    step 3 write_file -> written: true, path: agentloop-<run>-step-3.txt

With `AGENTLOOP_XDEV_OFF=1` the same run reports `ran: false` / `written:
false` and names the missing executor. Against the real xdev binary the
client reaches the ready frame and Start then fails on that machine's
malformed models.yml — which is the correct shape: Start fails, and the
server logs a warning and runs without a sandbox rather than dying.

## Checks

`make check` green: gofmt, vet, 14/14 packages (7 new tests in
internal/xdev, 3 new in internal/tools), lint 0 issues, PRD OK, selftest
12/12.

Docs: PRD §4 marks three tools built and one stub; §13 records this; USAGE's
config table gains the three variables and §9 splits what reads/writes today
from what still does not.
@linhdmn
linhdmn merged commit a33e428 into main Sep 21, 2026
1 of 3 checks passed
@linhdmn
linhdmn deleted the feat/xdev-rpc branch September 21, 2026 08:35
linhdmn added a commit that referenced this pull request Sep 21, 2026
…inters (#38)

The index said "the three remaining stub tools, the model-driven planner and
tier-combo names are the next milestone". Three of those four are now landed
(#33 execution, #34 the reasoner, #35 tier routing), so the note was stale and
the real next step was unnamed.

Rewritten as a handoff, with each item verified against the code rather than
recalled:

  1. **The loop cannot read a file** — four tools, none of which returns file
     *content*. `query` returns graph elements with a 400-char excerpt and a
     rung; it is not a reader. So the reasoner locates parseConfig, learns it
     is in config.go:41, and cannot look at it — which is exactly why the live
     demo wrote `func parseConfig() {}`. `write_file`'s description already
     promises a "read twin" that does not exist.
  2. **The run has no answer** — `goal_met` stores its synthesis in
     `PartialSynthesis`, a field named for the bound case, and the model's
     rationale sits in a step's `why`.
  3. **Success criteria are prose** — `ps.Success` appears exactly once in the
     codebase: as prompt text in reason.go. Nothing evaluates it.
  4. `web_search` is the last stub (or should be deleted).
  5. The two gaps #37 recorded: reply-out unscreened, ~740ms/call unmetered.
  6. Planning is still a rule table (no longer blocking — the chooser carries).

Plus the working notes a new session loses time rediscovering: the worktree
rule, `make check` as the gate, the shell quirks (nohup for backgrounded
servers, pkill matching its own command line, ports 8080/9699 taken), and the
one that matters most — every defect this repo shipped lately was a check that
could not fail (#25, #29, #32, #37, and #8's checklist ticked on the author's
behalf). Verify by breaking the thing, not by watching it pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant