Skip to content

CI: add apt-install.sh for unification and robustness #11629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
padelsbach wants to merge 1 commit into
base: master
Choose a base branch
from
Open

CI: add apt-install.sh for unification and robustness #11629

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 4 additions & 30 deletions .github/actions/ccache-setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,36 +62,10 @@ runs:
install -y --no-install-recommends ccache; then
echo "ccache installed offline from the staged .deb bundle"
else
# Same defence in depth as install-apt-deps: Acquire timeouts drop
# a stalled connection, `timeout` hard-kills a wedged apt-get, and
# only then does the retry loop get a non-zero exit to act on.
# 60s+90s x2: ccache is a ~700 KB package, and this loop has no
# budget input of its own, so it has to stay small enough to chain
# after install-apt-deps inside a 10-minute job.

# No wolfSSL job installs from the runner's Google/Microsoft apt repos,
# and a bad index on either fails apt-get update for everyone. Drop them.
grep -rlE 'dl\.google\.com|packages\.microsoft\.com' \
/etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -vf || true
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
ok=""
for i in 1 2; do
sudo dpkg --configure -a >/dev/null 2>&1 || true
if sudo DEBIAN_FRONTEND=noninteractive timeout -k 10 60 \
apt-get "${APT_OPTS[@]}" update -q && \
sudo DEBIAN_FRONTEND=noninteractive timeout -k 10 90 \
apt-get "${APT_OPTS[@]}" install -y \
--no-install-recommends ccache; then
ok=1
break
fi
echo "::warning::ccache apt install failed (attempt $i/2)"
# No sleep after the last attempt - this loop is not budgeted by
# the caller and chains onto install-apt-deps in the same job.
[ "$i" -eq 2 ] || sleep 5
done
[ -n "$ok" ] || { echo "::error::could not install ccache"; exit 1; }
# Short limits: this follows install-apt-deps in 10-minute jobs.
"$GITHUB_ACTION_PATH/../../scripts/apt-install.sh" --tries 2 \
--update-timeout 60 --install-timeout 90 --drop-vendor-sources \
--no-install-recommends ccache
fi
elif [ "${{ runner.os }}" = "macOS" ]; then
brew install ccache
Expand Down
93 changes: 23 additions & 70 deletions .github/actions/install-apt-deps/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ inputs:
Nominal wall-clock for the whole retry loop, split across the attempts
as per-command deadlines, so a wedged mirror is reported by this action
instead of the job being cancelled around it. The loop overshoots it by
retry-delay plus 10s of SIGKILL grace, and the per-command floors make
values below retries*80 inert. This, plus pull-timeout, plus any
ccache-setup in the same job, has to fit the caller's timeout-minutes -
the defaults need ~16 minutes.
the retry delays plus up to 20s of SIGKILL grace per attempt, and the
per-command floors make values below retries*80 inert. This, plus
pull-timeout, plus any ccache-setup in the same job, has to fit the
caller's timeout-minutes - the defaults need ~16 minutes.
required: false
default: '600'
pull-timeout:
Expand All @@ -37,11 +37,12 @@ inputs:
description: >
Tag of a prebuilt .deb bundle published to
ghcr.io/wolfssl/wolfssl-ci-debs by the ci-deps-image workflow
(e.g. "ubuntu-24.04-minimal"). When set, the packages are installed
from that bundle with no network access at all and the apt path below
is skipped. The install is all-or-nothing, so any failure - bundle
missing, not public, or not covering every requested package - falls
back to the apt path. Always safe to set; leave empty to use apt only.
(e.g. "ubuntu-24.04-minimal"). When set, on x64 runners, the packages
are installed from that bundle with no network access at all and the
apt path below is skipped. The install is all-or-nothing, so any
failure - bundle missing, not public, or not covering every requested
package - falls back to the apt path. Always safe to set; leave empty
to use apt only.
.github/scripts/check-ci-deps.py checks on every PR that the tag exists
and carries every package named above.
required: false
Expand Down Expand Up @@ -78,7 +79,8 @@ runs:
# packages).
- name: Install from the ghcr .deb bundle (offline)
id: ghcr
if: inputs.ghcr-debs-tag != ''
# Bundles hold amd64 .debs, so other runners go straight to apt.
if: inputs.ghcr-debs-tag != '' && runner.arch == 'X64'
shell: bash
run: |
set -u
Expand Down Expand Up @@ -200,67 +202,18 @@ runs:
if: steps.ghcr.outputs.satisfied != 'true'
shell: bash
run: |
RETRIES=${{ inputs.retries }}
DELAY=${{ inputs.retry-delay }}
BUDGET=${{ inputs.budget-seconds }}
# Update gets half of each attempt (20s..90s, apt's 3 x 30s ladder).
PER=$(( ${{ inputs.budget-seconds }} / ${{ inputs.retries }} ))
UPD=$((PER / 2))
[ "$UPD" -le 90 ] || UPD=90
[ "$UPD" -ge 20 ] || UPD=20
INS=$((PER - UPD))
[ "$INS" -ge 40 ] || INS=40
NO_REC=""
if [ "${{ inputs.no-install-recommends }}" = "true" ]; then
NO_REC="--no-install-recommends"
fi

# A wedged mirror hangs apt rather than failing it, so the retry loop
# below never fired and the job burned its whole budget instead.
# Defend in depth: apt drops a stalled connection after 30s and retries
# it (Acquire timeouts - this is what actually detects a wedge, in
# ~90s), `timeout` hard-kills an apt-get that wedged outside its own
# I/O loop, then the loop re-runs - re-reading apt-mirrors.txt, so a
# retry can land on a different mirror. apt resumes from
# archives/partial/, so a killed transfer is not restarted from
# scratch.

# No wolfSSL job installs from the runner's Google/Microsoft apt repos,
# and a bad index on either fails apt-get update for everyone. Drop them.
grep -rlE 'dl\.google\.com|packages\.microsoft\.com' \
/etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -vf || true
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)

DEADLINE=$(($(date +%s) + BUDGET))

# sudo resets the environment, so DEBIAN_FRONTEND has to ride along
# on each privileged command rather than being exported once.
for i in $(seq 1 $RETRIES); do
# Split what is LEFT over the attempts still to come, rather than
# slicing the budget up front: an attempt that ends early hands its
# remainder to the next one instead of dropping it.
#
# update gets half of an attempt, capped at 90s because that is what
# apt's own Acquire ladder (3 retries x a 30s timeout) needs to work
# through a stalled mirror and move on; install gets the rest. A
# sixth of an attempt used to be 50s of the sssd job's 600s budget,
# so update was killed mid-transfer twice and the job failed with
# 80% of its budget unspent. The floors keep a small budget usable;
# they are what makes it overshoot.
PER=$(( (DEADLINE - $(date +%s)) / (RETRIES - i + 1) ))
UPD=$((PER / 2))
[ "$UPD" -le 90 ] || UPD=90
[ "$UPD" -ge 20 ] || UPD=20
INS=$((PER - UPD))
[ "$INS" -ge 40 ] || INS=40
# A previous attempt killed mid-unpack leaves dpkg needing this.
sudo dpkg --configure -a >/dev/null 2>&1 || true
if sudo DEBIAN_FRONTEND=noninteractive timeout -k 10 $UPD \
apt-get "${APT_OPTS[@]}" update -q && \
sudo DEBIAN_FRONTEND=noninteractive timeout -k 10 $INS \
apt-get "${APT_OPTS[@]}" install -y \
$NO_REC ${{ inputs.packages }}; then
exit 0
fi
if [ "$i" -eq "$RETRIES" ] || [ "$(date +%s)" -ge "$DEADLINE" ]; then
echo "::error::apt-get failed after $i attempt(s) in ${BUDGET}s"
exit 1
fi
echo "::warning::apt-get failed (attempt $i/$RETRIES), retrying in ${DELAY}s..."
sleep $DELAY
DELAY=$((DELAY * 2))
done
"$GITHUB_ACTION_PATH/../../scripts/apt-install.sh" \
--tries ${{ inputs.retries }} --delay ${{ inputs.retry-delay }} \
--update-timeout "$UPD" --install-timeout "$INS" \
--drop-vendor-sources $NO_REC ${{ inputs.packages }}
6 changes: 6 additions & 0 deletions .github/ci-deps/packages-ubuntu-24.04-full.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ autoconf
autoconf-archive
automake
autopoint
autotools-dev
bc
bison
bubblewrap
build-essential
ccache
Expand All @@ -20,12 +22,14 @@ g++-10
g++-11
g++-12
g++-9
g++-multilib
gcc-10
gcc-11
gcc-12
gcc-9
gcc-multilib
gettext
git
gyp
jq
krb5-admin-server
Expand Down Expand Up @@ -88,9 +92,11 @@ ninja-build
pkg-config
pkgconf
psmisc
python3
python3-docutils
python3-impacket
python3-ldb
python3-pip
python3-psutil
python3-yaml
shellcheck
Expand Down
105 changes: 105 additions & 0 deletions .github/scripts/apt-install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/bin/sh
# Install apt packages with retries and hard timeouts, so a stuck mirror fails
# an attempt instead of hanging the job.
#
# usage: apt-install.sh [options] [--] [apt-get install args...]
# --tries N attempts (default 3)
# --update-timeout S limit for each apt-get update (default 90)
# --install-timeout S limit for each apt-get install (default 300)
# --delay S wait before the first retry, doubled after (5)
# --no-update skip apt-get update
# --drop-vendor-sources remove the runner's Google/Microsoft apt sources
# --warn-only report the last failure as a warning, not an error
# The remaining args go to apt-get install as is, so --no-install-recommends
# and --download-only work. With none, only apt-get update runs.
set -u

tries=3
update_timeout=90
install_timeout=300
delay=5
update=1
drop_vendor=0
level=error

while [ $# -gt 0 ]; do
case $1 in
--tries) tries=$2; shift ;;
--update-timeout) update_timeout=$2; shift ;;
--install-timeout) install_timeout=$2; shift ;;
--delay) delay=$2; shift ;;
--no-update) update=0 ;;
--drop-vendor-sources) drop_vendor=1 ;;
--warn-only) level=warning ;;
--) shift; break ;;
*) break ;;
esac
shift
done

for n in "$tries" "$update_timeout" "$install_timeout" "$delay"; do
case $n in
''|*[!0-9]*) echo "apt-install.sh: not a number: '$n'" >&2; exit 2 ;;
esac
done
for n in "$tries" "$update_timeout" "$install_timeout"; do
[ "$n" -gt 0 ] || {
echo "apt-install.sh: must be positive: '$n'" >&2
exit 2
}
done

export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" -eq 0 ]; then
as_root() { "$@"; }
else
# sudo resets the environment, so DEBIAN_FRONTEND has to go through it.
as_root() { sudo DEBIAN_FRONTEND=noninteractive "$@"; }
fi

apt_get() {
limit=$1
shift
as_root timeout -k 10 "$limit" apt-get -o Acquire::Retries=3 \
-o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30 "$@"
}

attempt() {
# An attempt killed mid-unpack leaves dpkg needing this.
as_root dpkg --configure -a || true
if [ "$update" -eq 1 ]; then
step=update
apt_get "$update_timeout" update -q || return
fi
[ $# -gt 0 ] || return 0
step=install
apt_get "$install_timeout" install -y -q "$@"
}

if [ "$drop_vendor" -eq 1 ]; then
# No job uses these, and a bad index on either fails apt-get update.
grep -rlE 'dl\.google\.com|packages\.microsoft\.com' \
/etc/apt/sources.list.d/ 2>/dev/null |
while read -r f; do as_root rm -vf "$f"; done
fi

i=1
while :; do
rc=0
attempt "$@" || rc=$?
[ "$rc" -ne 0 ] || exit 0
# A killed apt-get prints nothing, so name the timeout.
case $rc in
124|137) why="timed out" ;;
*) why="failed with exit code $rc" ;;
esac
[ "$i" -lt "$tries" ] || break
echo "::warning::apt-get $step $why (attempt $i/$tries)," \
"retrying in ${delay}s"
sleep "$delay"
delay=$((delay * 2))
i=$((i + 1))
done

echo "::$level::apt-get $step $why (attempt $i/$tries), giving up${*:+ on: $*}"
exit "$rc"
23 changes: 5 additions & 18 deletions .github/scripts/download-deb-closure.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,36 +16,23 @@ set -uo pipefail
LIST=${1:?package list}
DEST=${2:?destination directory}

APT_INSTALL="$(dirname "$0")/apt-install.sh"

mapfile -t PKGS < <(grep -vE '^[[:space:]]*#|^[[:space:]]*$' "$LIST")
echo "Packages (${#PKGS[@]}): ${PKGS[*]}"
export DEBIAN_FRONTEND=noninteractive
# Not rm -rf: in the container this directory is a bind mount.
mkdir -p "$DEST" && rm -f "$DEST"/*.deb
apt-get clean
# No wolfSSL job installs from the runner's Google/Microsoft apt repos, and a
# bad index on either fails apt-get update for everyone. Drop them. Already
# root here, so no sudo; the container images carry neither repo, so this is a
# no-op there.
grep -rlE 'dl\.google\.com|packages\.microsoft\.com' \
/etc/apt/sources.list.d/ 2>/dev/null | xargs -r rm -vf || true
# A single stalled mirror connection once hung -full for ~20 min (it normally
# finishes in a few). retry() only re-runs on a non-zero exit, so a hang never
# tripped it. Defend in depth: apt drops a stalled connection after 30s and
# retries it (Acquire timeouts), `timeout` hard-kills a wedged apt-get, then
# retry() re-runs from scratch.
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; }
retry timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q
"$APT_INSTALL" --tries 5 --update-timeout 120 --drop-vendor-sources
# Download each package's closure independently (requested package + any
# dependency not already installed) without installing. Per package, not one
# resolve of the whole list, so one unbundleable package - e.g. a conflict in
# the big -full union - cannot abort the rest; install-apt-deps falls back to
# apt for anything missing.
skipped=0
for pkg in "${PKGS[@]}"; do
retry timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y --download-only "$pkg" \
|| { echo "::warning::could not download $pkg"; skipped=$((skipped+1)); }
"$APT_INSTALL" --tries 5 --no-update --warn-only --download-only "$pkg" \
|| skipped=$((skipped+1))
done
cp /var/cache/apt/archives/*.deb "$DEST/" 2>/dev/null || true
# The steps that index and package these run unprivileged, and in the
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/afalg.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ jobs:
done
if [ -n "$missing" ]; then
grep -rn 'algif_' /etc/modprobe.d /lib/modprobe.d || true
sudo apt-get update -qq || true
sudo apt-get install -y "linux-modules-extra-$(uname -r)" || true
.github/scripts/apt-install.sh --tries 2 --warn-only \
--drop-vendor-sources "linux-modules-extra-$(uname -r)" || true
for m in $missing; do
if sudo modprobe --ignore-install "$m"; then
echo "modprobe $m: ok after modules-extra"
Expand Down
Loading
Loading