refactor(native): Ubuntu/RHEL backend — no nixos-infect (#154 Phase 4) - #325
Draft
defangdevs wants to merge 3 commits into
Draft
refactor(native): Ubuntu/RHEL backend — no nixos-infect (#154 Phase 4)#325defangdevs wants to merge 3 commits into
defangdevs wants to merge 3 commits into
Conversation
First piece of Phase 4: where the software comes from when there is no
system closure. `nix profile install github:defangdevs/agent-box/<rev>#runtime`
gets a distro host the same payloads, tools, agent CLIs and shared assets
the NixOS backend puts in its closure — atomically switchable and
`nix profile rollback`-able, with the distro package manager left to base-OS
patching only.
nix/runtime.nix builds every payload from modules/src/*, the same assets
bin/assemble-module.py embeds into the module, via builtins.readFile — so no
Nix-string escaping stands between the two backends and a payload cannot
drift by way of a quoting difference. Phase 2 is what makes this possible:
the payloads take their configuration from AGENT_BOX_* env and resolve
binaries from PATH, so neither backend owns them.
Layout the native host layer consumes:
bin/ unit-driven payloads + tools + agent CLIs
bin/agent-box-*-bare interactive CLIs, minus the env prelude that
`agentbox apply` will generate per host
share/agent-box/units/ the %i template units, verbatim
share/agent-box/caddy/ Caddyfile fragments (@token@ templates)
share/agent-box/guides/ default AGENTS.md sources
share/agent-box/web/ settings page assets
libexec/agent-box/ password-helper.py — generated per user, since
it crosses sudo and must not take paths from env
New check `runtime-profile` asserts each shipped payload is its modules/src
file verbatim, that the shared assets are byte-identical, that every script
parses under bash -n, and that the tools a session's PATH needs are present.
That is the guard against the failure that would matter most here: a native
box quietly running a different supervisor than the one tests/golden locks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsqGhkpjsb4kKAdVz25xKN
Second piece of Phase 4: what renders the host state when there is no
option tree and no nixos-rebuild. `agentbox apply` reads
/etc/agent-box/config.yaml (or .json — cloud-init can write either) and
idempotently renders exactly what the module renders on NixOS:
- /etc/agent-box/units/<user>.env and the web/settings/webhook peers —
the AGENT_BOX_* contract Phase 2 established
- per-instance host drop-ins (absolute ExecStart, PATH, tool pins), the
shape #295 landed on after finding that a template-level drop-in never
merges into an instance
- the %i template units, installed byte-for-byte out of the profile
- sudoers.d, tmpfiles.d, the session seed, the AGENTS.md pointer
- the Caddyfile, bound from the same @token@ fragments, plus a native
caddy.service (the shared password helper reloads that unit by name)
- the per-user password helper, with its constants APPENDED exactly as
the module's generated tail does — including the one override the
native side needs, since the body's SYSTEMCTL default is a NixOS path
- the interactive CLI wrappers, which cannot take their env from a unit
because they run from every PATH there is
`apply --first-boot` absorbs what the CFN UserData did inline: settle the
public IPv4, derive the sslip.io hostname, hash the web password with
argon2id (the same algorithm and stdin invocation the password helper uses
for a later change), and write the caddy env file. `agentbox signal` PUTs
the WaitCondition result, kept separate so a template signals only once the
box actually serves.
Testing, because a native box has no golden fixture to fall back on:
tests/test_agentbox.py renders into a tree (`apply --root`) and diffs it
against tests/native/expected — hermetic, from a fake profile built out of
modules/src, so it runs under a plain python3 with no Nix and no network.
It also asserts the units are installed verbatim, that a re-apply wants to
rewrite nothing, that JSON and YAML configs render identically, and that
bad configs are rejected. File modes are recorded in expected-modes.json
rather than as the fixture's own permissions: the fixture reaches CI
through the Nix store, which normalizes every file to 0444.
New check `agentbox-render` runs that suite and then hands the rendered
Caddyfile to `caddy validate` with a real argon2id hash — the one thing a
byte fixture cannot tell you, that the config actually adapts and
provisions. It found two real bugs already: @WEBHOOK_PATH@,
@WEBHOOK_SOCKET@, @DOWNLOADS_DIR@ and @TTYD_PORT@ were unbound (the
renderer now fails loudly on any unbound token), and the password helper
took its constants in the wrong order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PsqGhkpjsb4kKAdVz25xKN
Third piece of Phase 4: the 1-click that boots Ubuntu and stays Ubuntu. aws/lightsail-native-template.yaml installs Nix as a package manager, installs the pinned runtime profile, writes /etc/agent-box/config.yaml and runs `agentbox apply --first-boot`, then signals the WaitCondition once the box actually serves. No infection, no relabel, no reboot: the timeout drops from 2700 s to 1200 s, nano_3_0 (1 GiB) becomes a usable bundle, and the console's browser SSH keeps working because nothing lustrates /home. Two corrections to the Phase 4 design, both from the live launch that diagnosed the infect template's failure: 1. The payload CANNOT be the "#cloud-config" the design sketched. Lightsail prepends its own #!/bin/sh preamble to the launch script, so the payload is concatenated INTO a shell script — a YAML document would never be parsed as one. It stays a shell script here; a native EC2 template can use #cloud-config, since nothing is prepended there. 2. For the same reason the script re-execs itself under bash on its first line. This is what the infect template needed and never had. scripts/check_lightsail_userdata.py (salvaged from the closed #324) enforces that guard: it fails if the guard is missing, if a bashism appears before it, or if the prefix does not parse under dash -n. aws-ci.yml now cfn-lints the new template, runs that check, and runs the native renderer's tests. Not covered: the end-to-end launch has never run on live hardware. The README records that, and where to read the bootstrap log when a stack times out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsqGhkpjsb4kKAdVz25xKN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 4 of #154: agent-box on a stock Ubuntu/RHEL host, so a Lightsail launch needs no
nixos-infect, no reboot, no root-fs relabel, and 1 GiB bundles become viable. Draft — stacked on #295 (Phase 3), which must land first; review that one before this.Context for the timing: the infect-based 1-click has never actually booted. Lightsail prepends its own
#!/bin/shpreamble to the launch script, so the bootstrap ran under dash and died onset -o pipefail19s into first boot — diagnosed on live hardware, written up in #324. We are not fixing that path; this replaces it.Landed
1.
nix/runtime.nix+packages.runtime— where software comes from with no system closure:Payloads are built from
modules/src/*withbuiltins.readFile— the same assets the module embeds, with no Nix-string escaping in between. Phase 2 is what makes it work: payloads readAGENT_BOX_*env and resolve binaries from PATH, so neither backend owns them. Layout:bin/(payloads, tools, agent CLIs,agentbox,agent-box-python),bin/agent-box-*-bare(interactive CLIs minus the env prelude),share/agent-box/{units,caddy,guides,web},libexec/agent-box/password-helper.py.2.
bin/agentbox— the native config layer.agentbox applyreads/etc/agent-box/config.yaml(or.json) and idempotently renders the per-user env files, the per-instance host drop-ins, the%iunits byte-for-byte out of the profile, sudoers.d, tmpfiles.d, the session seed, the Caddyfile (same@TOKEN@fragments), a nativecaddy.service, the per-user password helper, and the interactive CLI wrappers; thenuseradd/systemctl enablewhat the config declares and disable instances for users removed from it.apply --first-bootabsorbs the CFN inline logic (settle the public IPv4, derive the sslip.io hostname, hash the password with argon2id, write caddy's env file);agentbox signalPUTs the WaitCondition result, separate so a template signals only once the box serves.3.
aws/lightsail-native-template.yaml— install Nix, install the profile, write the config, apply, signal. Timeout 2700 s → 1200 s;nano_3_0is the new default bundle; the console's browser SSH keeps working because nothing lustrates/home.Two corrections to the design in this issue
#cloud-config. On Lightsail the launch script is concatenated into a shell script, so a YAML document would never be parsed as one. It stays a shell script here; a native EC2 template can use#cloud-config.ExecSearchPathwith a bareExecStartis not used — the native drop-ins use absolute paths per instance, matching what refactor(module): systemd template units, %i = user (#154 Phase 3) #295 found in practice.Tests
agentbox-render(new check):tests/test_agentbox.pyrenderstests/native/config.{json,yaml}into a tree and diffs it againsttests/native/expected; asserts units are installed verbatim, a re-apply rewrites nothing, JSON and YAML agree, bad configs are rejected. Then it hands the rendered Caddyfile tocaddy validatewith a real argon2id hash — the thing a byte fixture cannot tell you. It caught two real bugs already: four unbound Caddyfile tokens (@WEBHOOK_PATH@,@WEBHOOK_SOCKET@,@DOWNLOADS_DIR@,@TTYD_PORT@— the renderer now fails loudly on any unbound token), and the password helper's constants appended in the wrong order.runtime-profile(new check): every shipped payload must be itsmodules/srcfile verbatim, assets byte-identical, scriptsbash -n-clean, PATH tools present.scripts/check_lightsail_userdata.py(salvaged from fix(lightsail): re-exec the launch script under bash — the 1-click has never booted #324): fails on a missing bash re-exec guard, a bashism ahead of it, or a prefix that will not parse underdash -n.modules/src). File modes live inexpected-modes.json, because the fixture reaches CI through the Nix store, which normalizes every file to 0444.Green locally on aarch64:
runtime-profile,agentbox-render,golden-snapshot,module-generated-up-to-date, plusaws cloudformation validate-templateon the new template.Still to come
deploy-test.yml(needslightsail:*on the test account's OIDC role, which this repo does not manage).agent-web-auth-secretsequivalent for password rotation is handled by the shared helper; the singleton's remaining manifest rework (design item 5) is still open.config.yamluses, and diff both renders in one check.🤖 Generated with Claude Code
https://claude.ai/code/session_01PsqGhkpjsb4kKAdVz25xKN