fix(runners): guard SEGMENT against set -u in start-runner.sh - #5233
Open
jensenbox wants to merge 1 commit into
Open
fix(runners): guard SEGMENT against set -u in start-runner.sh#5233jensenbox wants to merge 1 commit into
jensenbox wants to merge 1 commit into
Conversation
`start-runner.sh` assigns `SEGMENT` only inside the X-Ray branch (line 191) but
consumes it unconditionally at line 263 and in the `error_handler` trap (line
98). With `enable_ssm_on_runners`-style tracing disabled — the default — the
branch never runs and `SEGMENT` is never assigned.
That is harmless while user-data runs without `set -u`. It becomes fatal as soon
as anything enables `-u` earlier in the same shell, and user-data is one
concatenated script: `${post_install}` is spliced inline at
`modules/runners/templates/user-data.sh:63`. A consumer whose `post_install`
hook opens with `set -euo pipefail` therefore aborts user-data here — after the
runner has registered with GitHub, but before its listener starts.
The failure mode is unusually quiet: the runner appears healthy in the GitHub
UI (it registered and holds an agent ID), but never claims a job, so scale-down
reaps it as idle and the scaler launches an identically-broken replacement. We
lost every job on one runner pool for ~19h before tracing it to:
√ Runner successfully added
Tagging instance with GitHub runner agent ID: 49534
/var/lib/cloud/instance/scripts/part-001: line 703: SEGMENT: unbound variable
ERROR: runner-start-failed with exit code 1 occurred on 1
FAILED Failed to start cloud-final.service - Cloud-init: Final Stage.
Use `"$${SEGMENT:-}"` at both unguarded call sites. This restores the script's
existing intent rather than changing behaviour — both helpers already open with
local SEGMENT_DOC="$1"
if [ -z "$SEGMENT_DOC" ]; then
echo "No segment doc provided"
return
fi
so they were written to tolerate an absent segment; the call sites simply never
expressed it in a `-u`-safe way. Line 192's `echo "$SEGMENT"` is deliberately
left alone: it sits inside the branch that assigns the variable.
Note the `$$` — this template is itself rendered through `templatefile()`, so a
bare `${SEGMENT:-}` is parsed as a Terraform interpolation and fails the plan
with "Template interpolation doesn't expect a colon at this location". Existing
precedent in the same file: `$${extra_flags}` and `$${config}`.
Verified by rendering the template with `templatefile()` and running `bash -n`
over the output.
jensenbox
added a commit
to closient/terraform-aws-github-runner
that referenced
this pull request
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
modules/runners/templates/start-runner.shassignsSEGMENTonly inside the X-Ray branch (line 191) but consumes it unconditionally at line 263 and in theerror_handlertrap (line 98). With tracing disabled — the default — the branch never executes andSEGMENTis never assigned.That is harmless while user-data runs without
set -u. It becomes fatal the moment anything enables-uearlier in the same shell, and user-data is one concatenated script:${post_install}is spliced inline atmodules/runners/templates/user-data.sh:63. So a consumer whosepost_installhook opens withset -euo pipefail— a perfectly ordinary thing for a hook to do — silently arms this latent bug for the module's own script.Why it is worth fixing rather than documenting
The failure mode is unusually quiet. User-data dies after the runner registers with GitHub but before its listener starts, so:
From the outside it looks like a capacity or scaling problem, not a script error. We lost every job on one runner pool for ~19 hours before finding it in the instance console:
Fix
"$${SEGMENT:-}"at both unguarded call sites.This restores the script's existing intent rather than changing behaviour. Both helpers already begin with:
so they were written to tolerate an absent segment doc — the call sites simply never expressed it in a
-u-safe way. With tracing enabled nothing changes; with tracing disabled the helpers now hit the early return they already had.Line 192's
echo "$SEGMENT"is deliberately untouched: it sits inside the branch that assigns the variable.Note the doubled
$— this template is itself rendered throughtemplatefile(), so a bare${SEGMENT:-}is parsed as a Terraform interpolation and fails the plan with "Extra characters after interpolation expression; Template interpolation doesn't expect a colon at this location". Existing precedent in the same file:$${extra_flags}and$${config}(lines 257, 269).Verification
templatefile()and confirmed the output contains"${SEGMENT:-}"at both sites.bash -nover the rendered script.SEGMENT: unbound variablenow reach a running listener, claim jobs, and complete them.Scope
Two characters at two call sites. No behaviour change for tracing-enabled consumers.