fix(ci): rebuild cluster host keys per run so a node rebuild self-heals - #247
Merged
Conversation
msa2-client was reinstalled onto a new SSD on 2026-08-18 and so presents a new
SSH host key. The inventory sets StrictHostKeyChecking=accept-new, which accepts
UNKNOWN hosts but correctly REFUSES CHANGED ones, so every subsequent scheduled
run died:
fatal: [msa2-client]: UNREACHABLE!
Host key verification failed.
msa2-server : ok=277 msr1 : ok=48 msa2-client : unreachable=1
That took out both the Nightly Validation and the weekly rated Benchmark Tier.
The symptom is badly misleading: a perfectly healthy machine (19h uptime, .195
up on the bond, on the tailnet) looks dead. The stale entry lives in the
persistent ~/.ssh/known_hosts of whichever host ran the job, and because Ubuntu
hashes known_hosts by default `grep` finds nothing — only `ssh-keygen -F`
reveals it, which cost real time during diagnosis.
Fix: scan the cluster fresh into a per-run known_hosts under the runner root
(on /tmp, i.e. tmpfs, discarded on reboot) and point ansible at it via
UserKnownHostsFile. A rebuilt node is picked up automatically; host-key checking
still applies WITHIN the run; and the operator's persistent known_hosts is never
mutated. Unset outside CI, so dev-machine behaviour is unchanged.
The host list comes from ansible/inventory.yml, keeping one source of truth for
topology — adding a node needs no workflow edit.
Verified end-to-end against the live cluster:
- empty known_hosts -> all 3 hosts SUCCESS, 3 keys learned into the file
- well-formed WRONG key planted for .195 -> UNREACHABLE, "REMOTE HOST
IDENTIFICATION HAS CHANGED" (proves the file is used for verification,
not merely written)
- correct scanned keys -> SUCCESS
- script runs on bash 3.2 (no mapfile, no nested heredocs) so it stays
testable on the dev Mac, not just the Linux runners
Non-obvious detail found while testing: ControlMaster multiplexing BYPASSES
host-key verification entirely, so a negative test against an already-open
connection passes misleadingly. The sockets must be cleared to test this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
msa2-client was reinstalled onto a new SSD and presents a new SSH host key.
StrictHostKeyChecking=accept-newaccepts unknown hosts but correctly refuses changed ones, so every scheduled run after the rebuild died:That took out both Nightly Validation and the weekly rated Benchmark Tier.
The symptom is badly misleading — a healthy machine (19 h uptime,
.195up on the bond, on the tailnet) looks dead. The stale entry sits in the persistent~/.ssh/known_hostsof whichever host ran the job, and Ubuntu hashesknown_hostsby default, sogrepfinds nothing. Onlyssh-keygen -Freveals it.Fix
Scan the cluster fresh into a per-run
known_hostsunder the runner root (/tmp, tmpfs, gone on reboot) and point ansible at it withUserKnownHostsFile.known_hostsis never mutated~/.ssh/known_hosts, dev-machine behaviour unchangedHost list comes from
ansible/inventory.yml, so topology has one source of truth and adding a node needs no workflow edit.Verified against the live cluster
known_hosts.195REMOTE HOST IDENTIFICATION HAS CHANGEDThe negative test is the important one — it proves the file is used for verification, not merely written.
Script is bash-3.2 compatible (no
mapfile, no nested heredocs) so it stays testable on the dev Mac, not only on the Linux runners.Non-obvious finding
ControlMastermultiplexing bypasses host-key verification entirely. My first two negative tests passed misleadingly because ansible reused an already-open socket. The control sockets have to be cleared to test this at all — worth knowing for any future host-key debugging.Security note
This is trust-on-first-use per run rather than persistent pinning. For a private-LAN bench cluster that's the right trade: the alternative is that any node rebuild silently wedges every scheduled run, and presents as hardware failure.