Spec: two-host production, and heavy-model modes on top of it - #344
Open
kh0pper wants to merge 10 commits into
Open
Spec: two-host production, and heavy-model modes on top of it#344kh0pper wants to merge 10 commits into
kh0pper wants to merge 10 commits into
Conversation
added 10 commits
September 9, 2026 18:06
Kevin's chosen standing configuration, from three sessions converging tonight: normal production bots on crow, Qwen3.8-Flash-Next at 1M on raven as normal production, and the option to evict production on both boxes to run either GLM-5.3-Flash or DSv4-Flash two-box. Docs only. No code, no catalog edits. Implementation ownership stays with the crow side for the format, orchestrator routing and catalog; the two harness items are pi-lab's. Why this shape works, with the measured numbers: Flash-Next Q4_K_XL is the only one of the three heavies whose weights leave room for a 1M KV on a single box (about 106 GiB of 124, 17.7 GiB free on raven), and single-box on raven beats the two-box split on both prefill and decode, so the 1M model needs one box and crow stays untouched. GLM IQ4_XS at 146 GiB and DSv4 IQ4_XS at 128 GiB exceed one box outright, so they are structurally two-box and cannot join the standard state; they get their own mode with both hosts evicted. The design also sidesteps the harness's biggest limitation. A persistent raven service needs nothing to start it on demand, and the harness can already consume a remote endpoint over the LAN even though it cannot lifecycle one. Carries pi-lab's five window properties verbatim as requirements, plus its veto that a curated catalog must never offer a known-wedge shape as a one-tap option, and crow-34's sequencing: serving.class first, then gates, then build, then topology. Records three open decisions for Kevin: retiring the crow-dsv4 provider row, the priority of the two harness changes against the catalog work, and 524k as a curated rung.
…dering constraint Folds in review from both other sessions. crow-34, as format/catalog owner, made two calls. serving.class is a catalog schema field on the model entry and must NOT live in settings.localModels: a curated safety property held in per-instance settings is one edit away from a wedge-risk shape being relabelled resident, and a veto needs git review. Instance settings may narrow what a box will run, never widen it. The group convention goes the other way, into settings.localModels, because group/evicts describe what else runs on a box rather than a property of a model. Its refinement, adopted: class is a property of a config, not a model, and the catalog has no variant concept yet. So the entry-level field is now defined as a ceiling rather than a description, the most dangerous supported shape for that model, with class moving onto the variant when topology lands and the entry value kept as an enforced ceiling. The ratchet is one-way by design. The group convention is now labelled a workaround, with a host-aware eviction relation named as the intended end state, so future multi-host entries do not re-learn the trap. pi-lab supplied a hard ordering constraint that was missing: the two-host window must land BEFORE Flash-Next production goes onto raven. dsv4-window.sh evicts crow prod only and its preflight requires raven idle under 2 GiB GTT residue, so once raven carries prod every two-box arm either fails preflight or contends with a live service. New 3.0. Also: crow prod measured at 61 to 62 GiB across two window teardowns, anchoring the 43 GiB bound for the 35b; port 8030 confirmed free on raven, closing an open item; the catalog memory fields described exactly (min_vram_mb absent at entry level, present-but-zero per quant, min_ram_mb populated and useful); and pi-lab's offer of an isolated per-container GTT measurement recorded as needing a deliberate go.
Kevin: "we might need to add the second box to the port registry if we haven't already." We hadn't. The registry described one machine throughout, with no host dimension anywhere, so under the two-host production design a port number no longer identifies an allocation. Adds a raven section covering 8030 (Flash-Next production), 8031-8033 (two-box masters, reserved) and crow:50052 (the RPC worker on the USB4 link). 8030 through 8033 verified free on raven, which listens only on 22, 53, 631 and two ephemeral ports. The first column is deliberately raven:<port> rather than a bare number, because check-port-allocation.js reads bare numbers in the first cell as crow allocations. Keeping raven rows unparseable to it is correct today and is a stopgap; making the checker host-aware is recorded as follow-up. Checker still passes: 45 ports, all documented, no new collisions. Two gaps found while writing it, recorded rather than silently fixed. Crow's own model ports 8003, 8006 and 8014 are absent from the table (only the copilot's 8010 is listed) because they bind from crow-addons composes, which the conventions already call a separate registry that nothing checks. And 8098 is allocated to searxng on crow's loopback while the two-box zoo arms use 8098 on raven: both correct today because they are different machines, and neither the table nor the checker can express that.
…ld catch pi-lab pointed out that raven:8098 has been bound by its zoo arms for days while the new raven section omitted it, which makes the registry look authoritative for raven while missing a port in regular use. That is worse than not covering raven at all, and it was right. Scanning pi-lab/scripts for bound ports rather than adding only the one named turned up three more: raven:8035 (the R24 result-check server on loopback), raven:50053 (a second ggml-rpc-server, for arms running two workers on raven's one GPU), and the fact that 50052 binds on BOTH machines depending on which is the worker, 10.99.0.1 for crow and 10.99.0.2 for raven. That last one is the clearest possible argument for the raven:<port> prefix, since a bare 50052 cannot say which box it means. Section now splits production and reserved from benchmark-transient, the latter being ports bound by scripts rather than compose files, so nothing would ever have caught them. Also records crow:8020 and crow:8021, absent from the main table for the same reason; 8020 matters because the crow-dsv4 provider row points at it. Checker unchanged: 45 ports, all documented, no new collisions.
pi-lab reviewed its own tree against the previous commit and found one wrong attribution, one live collision, and three more ports. Verifying those rather than accepting them found five more that neither of us had, and corrected pi-lab's correction. Fixed: 8021 is bound by 25 raven-* master scripts and is the standard two-box master port from R1 through R26, so it belonged on raven. But it is NOT raven-only as reported: four non-raven scripts default to it and all of them run as duties under dsv4-window.sh, which is the crow window. So 8021 binds on both boxes, a third instance alongside 8099 and 50052. Collision: raven-r23-knob-sanity.sh was already binding 127.0.0.1:8031, inside the range this section reserved for two-box masters. pi-lab moved it to 8037 (KS_PORT) rather than have the reservation move. Found by scanning invocations rather than assignments: crow:8022, 8024, 8025, 8026 and 8027, all DSv4 Vulkan hyper-connection duties. Added from pi-lab's read of its own tree: raven:8036 (the R25/R25b single-box stack, which is the Flash-Next shape chosen for production), crow:8012/8013 (GLM-5.2 windows) and crow:8023. The section now lists twenty-one script-bound ports across both hosts, none of which any compose file declares, and calls out the three that bind on both machines. It also records how the list was built, because three passes each found more and a human reading the tree still found what the patterns missed. Any scan of this kind is a lower bound. Checker unchanged: 45 ports, all documented, no new collisions.
method note that silently failed to land
Three things, one of them a correction to my own last commit.
The method note was never in the file. The previous commit's message
claimed the section records how the list was built; it does not,
because that edit used an unasserted string replace whose pattern did
not match, so it no-opped silently while every other edit in the same
script succeeded. Exactly the failure mode this section is about.
Every edit here asserts its match count first. The note is now
actually present.
Adds the constraint a port registry structurally cannot express: raven
cannot host 8030 and 8036 at the same time however free both ports
are, because they are the same Flash-Next single-box config and each
wants about 92.6 GiB on a 124 GiB box. Two free ports, one machine's
worth of memory. Recorded as the general case, since on a single-tenant
box port availability is necessary and never sufficient. Mirrored into
the spec's open items, because an R25-lineage arm must now treat the
production service as something to evict.
Adds the third scan form, which is the easiest to miss and the one
that caused the wrong host attribution: PORT="${PHASE0_PORT:-8021}"
matches no scan for PORT=8021, port 8021 or :8021. It is multiplying,
because parameterising a hardcoded port is the correct fix for the
ambiguities here and creates a new hiding place each time.
The method note now also names the two failure modes that actually
occurred, neither of which was a missing grep: a scan surfaced a
colliding port and its author read the hit as confirming their own
reservation, and a categorical negative was asserted from a pattern
that could not have matched the files in question.
Checker unchanged: 45 ports, all documented, no new collisions.
pi-lab hit a fourth instance one message after I reported the third, and it completes the pattern well enough to be worth generalising. The four, none of which was a missing grep: a trailing comment on a shell assignment line swallowed the assignments after it, passing bash -n and failing later under set -u; a categorical negative was asserted from a pattern that could not have matched the files; a string replace reported success without applying, because its pattern missed the file's line wrapping; and a grep confirming a correct edit returned zero, because it was single-line and the text wrapped. All four are one failure: the check and the thing checked disagree about shape, and the check returns the reassuring answer. The fourth is the worst, because it makes a correct edit look failed and the natural response is to apply it twice. Records three cheap habits: assert the match count before substituting so a no-op is loud, verify against text normalised the same way the edit was written, and never assert a categorical negative from a pattern scan, since the honest form is "this pattern found none". This commit's own verification was done on whitespace-flattened text for exactly the fourth reason. Checker unchanged: 45 ports, all documented, no new collisions.
pi-lab's generalisation of why its version of the raven note beat mine. "Evict raven production before an R25-lineage arm" is a rule someone forgets at 02:00; "92.6 GiB each on a 124 GiB box" is arithmetic anyone can redo and get the same answer. The second form outlives its authors and fails loudly when the hardware changes, where a remembered rule quietly stops being true. New 4.1, so later sections adding constraints have the preference stated rather than inferred.
shouldSyncRow, not SYNCED_TABLES membership, decides whether a providers row replicates, and it carries a loopback carve-out that gates both emit and apply. crow-dsv4 therefore never synced and had to be retired once per instance; the claim that it was a fleet decision was false and would have sent a reader looking for a sync bug that does not exist. The raven row sits on the other side of that carve-out. Record what follows: host rides the wire and must say raven everywhere, and the row should not be published before the endpoint is reachable from the instances that inherit it.
An earlier revision of 3.1 read as though the raven row's host='local' was an omission. It was a derivation: inferHost supplies a host when the caller omits one, and it treats the whole private address space plus Tailscale's CGNAT range as local. Framing it as a forgotten field invites a checklist; framing it as an unsound inference invites the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. One new spec, no code and no catalog edits. This is the artifact the crow side asked for before starting the catalog work: Kevin's direction plus a spec, rather than three sessions converging on a design in messages.
The configuration
Kevin's words: run the normal production bots on crow, run Qwen3.8-Flash-Next at 1M on raven as normal production, with the option to evict production on both boxes to run either GLM or DSv4. Three states, and only three.
The standard state is the point: a fast runner and a 1M-context model available at the same time, on different boxes, with no eviction, no RPC and no shared GPU.
Why this shape and not another
Measured, not assumed:
What it makes cheaper
The harness cannot start or stop a raven-hosted model at all:
startModelNowrunsdocker composelocally with no host field, no ssh and no docker context. A persistent raven service needs nothing to start it on demand, and the harness can already consume a remote endpoint becauseisRunningis an HTTP fetch over the LAN. So the standard state sidesteps that limitation entirely, which is the main reason this design is cheaper than the alternatives considered tonight.What it still needs
group/evictsare global and host-blind: raven's entry gets its own group andevicts: [], so it can never stop crow's 35b.setRoleModel's conflict test asks "are both local" rather than "would this evict that", so it argues against this configuration on every binding change.wouldEvictis already the right predicate.Carried through as requirements
pi-lab's five window properties, verbatim, each written after an incident, with the orchestrator calling
dsv4-window.shrather than reimplementing it: out-of-process deadman, live-state memory gate, verified restore, reservation file, two-strike guard. Plus its veto: a curated catalog must never be able to offer a known-wedge shape as a one-tap option.crow-34's sequencing, adopted:
serving.classfirst as the independently shippable piece that protects the box on day one, thengates, thenbuild, thentopology.A proposed answer to the open seam crow-34 raised, window expiry during a live turn: drain, never extend. The deadman keeps its absolute wall clock; a soft deadline before it stops accepting new turns and lets the in-flight one finish; if that turn outlives the hard deadline, the deadman wins. Extending on activity is rejected, because that is how a cap stops being a cap.
Open decisions recorded, not taken
crow-dsv4provider row: enabled, pointing at a dead port, validating inresolveModelso a bot fails every turn on connection refused. Live on both instances.providersis inSYNCED_TABLES, so no session has touched it. Under this spec DSv4 has no standing endpoint, so retiring the row looks right.Nothing was started, no box reservation taken, and the Thursday chain is untouched.