[SPARK-56808][INFRA][3.5] Fix branch-3.5 base image build against Ubuntu focal archive rotation#55785
Draft
LuciferYang wants to merge 2 commits intoapache:branch-3.5from
Draft
[SPARK-56808][INFRA][3.5] Fix branch-3.5 base image build against Ubuntu focal archive rotation#55785LuciferYang wants to merge 2 commits intoapache:branch-3.5from
LuciferYang wants to merge 2 commits intoapache:branch-3.5from
Conversation
…ntu focal archive rotation Three changes in dev/infra/Dockerfile: 1. Add mirrors.edge.kernel.org as an additional APT source (same approach as master), providing a stable fallback when archive.ubuntu.com / security.ubuntu.com rotate point-release packages out of the archive. 2. Merge `apt-get update` into the SPARK-39959 install step so its APT index is aligned with the archive at install time. 3. Bump FULL_REFRESH_DATE to 20260510 to invalidate the GH Actions base-image cache so this fix takes effect. The scheduled Base image build on branch-3.5 has been failing with 404s when fetching -dev packages (libtiff5-dev, libharfbuzz-dev, libglib2.0-dev, libfreetype6-dev, ...). focal entered ESM in April 2025 and point releases rotate out of the canonical archives faster than before. Example failure: https://github.com/apache/spark/actions/runs/25599925191/job/75152057946
… avoid CA trust failure on focal-20221019
The previous commit added the kernel.org mirror over HTTPS, but the
ubuntu:focal-20221019 base image ships with a stale CA bundle and no
ca-certificates is installed at the first `apt-get update`, causing:
Certificate verification failed: The certificate is NOT trusted.
W: No system certificates available. Try installing ca-certificates.
E: The repository 'https://mirrors.edge.kernel.org/ubuntu focal Release'
does not have a Release file.
Switch the mirror.list entries from https:// to http://. APT verifies
Release/Packages indices via GPG signatures and each .deb via SHA256,
so HTTP is safe here and unblocks `apt-get update` on this older base
image.
Member
|
FYI, I'm working on the same issue too with another approach. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Three small changes in
dev/infra/Dockerfileto make the scheduledBase image buildjob onbranch-3.5green again:https://mirrors.edge.kernel.org/ubuntuas an additional APT source (focal,focal-updates,focal-security). This mirrors the pattern themasterbranch already uses and provides a stable fallback whenarchive.ubuntu.com/security.ubuntu.comrotate point-release packages out of the canonical archive.apt-get updateinto theSPARK-39959install step so its APT index is aligned with the archive at install time, instead of relying on an index cached many Docker layers earlier.FULL_REFRESH_DATEfrom20221118to20260510so the GH Actions base-image cache is invalidated and this fix actually takes effect on the next run.The base image itself (
ubuntu:focal-20221019) is unchanged —branch-3.5is in maintenance and not a good place to upgrade tojammy.Why are the changes needed?
The scheduled
Build (branch-3.5, Scala 2.13, Hadoop 3, JDK 8)workflow on 2026-05-09 failed duringBase image buildwith multiple404 Not Founderrors while installing-devpackages (libtiff5-dev,libharfbuzz-dev,libglib2.0-dev,libfreetype6-dev,libblkid-dev,libmount-dev, ...). See:https://github.com/apache/spark/actions/runs/25599925191/job/75152057946
Root cause: Ubuntu 20.04 (focal) entered ESM in April 2025. Security point releases rotate out of
archive.ubuntu.com/security.ubuntu.comfaster than before. When the Dockerfile's cached APT index (fetched many layers earlier) references a point-release version that has since been rotated,apt-get installhits 404.The fix avoids the race by (a) adding a reliably-synced additional mirror and (b) refreshing the APT index right before the failing install step.
Does this PR introduce any user-facing change?
No. Infra-only change to the CI base image on
branch-3.5.How was this patch tested?