pilotctl: fix pidfile lifecycle — foreground crash no longer bricks restart#405
Merged
Conversation
…estart 2026-07-19 incident (pilot-director, Lambda box): a systemd unit running `pilotctl daemon start --foreground` crash-looped for HOURS with 'another daemon start is in progress (PID file locked)' after the daemon died overnight. Root cause chain: 1. The O_EXCL pidfile claim writes a '0' placeholder, and the --foreground path syscall.Exec's the daemon WITHOUT ever writing the real PID — so the file holds '0' for the daemon's whole life. 2. After any crash, the next start reads pid=0, skips the pid>0 staleness cleanup, and dies on the O_EXCL claim. Every systemd respawn hits the same wall: an unrecoverable restart loop that also silently defeats the rx-watchdog's exit-86 'supervisor will respawn us' recovery (#369/#387/#401). Fixes, all paths: - foreground: write os.Getpid() before exec (exec keeps the PID), so the real daemon PID is recorded; `daemon status` now reports systemd-managed daemons correctly instead of 'pid file stale'. - start: a pidfile with no usable PID (empty/garbage/'0') is never a live daemon — remove it instead of failing the claim. - new pidLooksLikeDaemon(): verifies the PID's executable base name is actually pilot-daemon/daemon (via /proc cmdline, ps fallback) — Signal(0) alone matches ANY process that inherited a recycled PID, which made start refuse ('already running') and stop SIGTERM/SIGKILL an innocent bystander. Used by the start gate, stop, and status. Tests: pidLooksLikeDaemon rejects self/sleep/dead pids; readPID treats '0'/empty/garbage as unusable while a real PID round-trips. Live repro verified: a poisoned '0' pidfile no longer blocks start. Full cmd/pilotctl suite green; gosec finding count unchanged vs main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // Stale PID file — clean up silently | ||
| // Stale PID file (dead daemon, or the PID was recycled by an | ||
| // unrelated process) — clean up silently. | ||
| os.Remove(pidFilePath()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident (2026-07-19, pilot-director)
A systemd unit running
pilotctl daemon start --foregroundcrash-looped for hours with "another daemon start is in progress (PID file locked)" after the daemon died overnight. The node was dark until manual intervention — and this failure mode silently defeats the rx-watchdog's exit-86 recovery (#369/#387/#401): the watchdog exits expecting the supervisor to respawn a clean daemon, but every respawn is rejected by the stale lock.Root cause
The O_EXCL pidfile claim writes a
"0"placeholder; the--foregroundpath thensyscall.Execs the daemon without ever writing the real PID. The file holds0forever. After any crash:readPID()==0→ thepid > 0staleness cleanup is skipped → the O_EXCL claim fails → fatal. Unrecoverable by the supervisor. (The director's pidfile was literally 2 bytes:0\n.)Fixes (all pidfile consumers)
os.Getpid()before exec — exec preserves the PID, so the real daemon PID is recorded. Side effect:daemon statusnow correctly reports systemd/launchd-managed daemons instead of "pid file stale"."0") is never a live daemon → remove and proceed.pidLooksLikeDaemon()verifies the executable's base name is actuallypilot-daemon/daemon(/proc/<pid>/cmdline,psfallback).Signal(0)alone matches any process that inherited a recycled PID — which madestartrefuse ("already running") and, worse, letstopSIGTERM/SIGKILL an innocent bystander. Used by start gate, stop, and status; fails safe (socket probe / socket-owner discovery remain the authoritative checks).Verification
pidLooksLikeDaemonrejects self/sleep/dead PIDs;readPIDtreats0/empty/garbage as unusable, real PID round-trips.0\npidfile →daemon startnow proceeds past the claim (previously: hard "PID file locked").cmd/pilotctlsuite green; gosec finding count unchanged vs main (19=19; newpscall is#nosecfixed-argv).🤖 Generated with Claude Code