debuggable installations in eic/containers: wire --debug-source and --debug-symbols into the dbg environment

[SebastianPaucar]

This PR adds the framework related to the following design discussion for debuggable installations in spack packages:

• PR 52768 (spack/spack): DWARF-referenced source hook and symbol splitting plus GDB init for debuggable installations.
• PR 19 (spack/compiler-wrapper): -ffile-prefix-map=<staging>=. and --build-id injection in compiler-wrapper.
• PR 5353 (spack/spack-packages): set SPACK_DEBUG_PREFIX_MAP for -ffile-prefix-map=<stage>=. injection at compiler-wrapper level.
• Spack Debuggable Installations Demo
  (SebastianPaucar/spack): --debug-source and --debug-symbols PoC container demo (verified end-to-end).

In summary, this PR adds --debug-source and --debug-symbols to spack install when ENV=dbg, via a new conditional in scripts/build-eic.sh that appends to SPACK_INSTALL_FLAGS.

[wdconinc]

@SebastianPaucar In master (#393) we added a mirror for the source that has the ssl certificate errors in CI here.

[SebastianPaucar]

@SebastianPaucar In master (#393) we added a mirror for the source that has the ssl certificate errors in CI here.

Branch updated with #393. Also implemented OCI push/autopush (per mirrors.yaml.in) and fetch of debug info (sources and symbols) into/from GHCR and eicweb mirrors (verified locally with my GHCR), according to this commit in my Spack branch feature/debuggable-installations-v1.2.2 #a5874e8. Working in the documentation.

[SebastianPaucar]

@SebastianPaucar In master (#393) we added a mirror for the source that has the ssl certificate errors in CI here.

Noticed some of the newer cherry-picks in spack-packages.sh were missing. Added.

[wdconinc]

It seems this PR has created a whole new spack build cache at https://eicweb.phy.anl.gov/containers/eic_container/container_registry/154 because we reuse the branch name as container registry name ;-)

[github-actions]

Capybara summary for PR 388

• rec_dis_10x100_minQ2=1000_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_10x100_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_18x275_minQ2=1000_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_18x275_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_5x41_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_5x41_minQ2=1_epic_craterlake_amd64_vs_arm64 (Δ: 4156)
• rec_dis_5x41_minQ2=1_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• rec_e_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_e_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 3713)
• rec_e_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• rec_pi_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_pi_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 3420)
• rec_pi_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_10x100_minQ2=1000_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_10x100_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_18x275_minQ2=1000_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_18x275_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_5x41_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_5x41_minQ2=1_epic_craterlake_amd64_vs_arm64 (Δ: 395)
• sim_dis_5x41_minQ2=1_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• sim_e_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_e_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 395)
• sim_e_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• sim_pi_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_pi_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 328)
• sim_pi_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
  ^{_{Last updated 2026-09-18T20:41-04:00 02cf275}}

[SebastianPaucar]

Hi @wdconinc, 84 checks have now successfully passed. Can you please rerun the remaining failing check (eicweb/eic_container)?

On another note, I figured out the root cause of the previous SSL issues. dawncut has buildcache coverage, while dawn does not, and their host URL (http://geant4.kek.jp/~tanaka/src/) is broken, as stated in eic-spack/spack_repo/eic/packages/dawncut/package.py. These are not related to our debuggable framework. I dropped both from xl/spack.yaml and just opened a dedicated PR if useful here #407.

[SebastianPaucar]

Hi @wdconinc, since the dawn and dawncut errors have gone, I think we are ready to move forward with the implementation of the debuggable spack installations framework.

I want to flag that the source and symbols capture only produces anything useful when the package is actually built with debug info (by adding build_type=Debug, -g, build_type=RelWithDebInfo, or any equivalent flags in the specs), so the framework can strip out the debug artifacts from it. Right now, this PR applies the framework to the dbg env seamlessly, since the specs contain build_type=Debug.

Since xl currently builds with no debug flags, adding --debug-source --debug-symbols to xl's installation stage wouldn't capture anything meaningful. It'd need xl to build with debug flags. The cost is that xl has much more specs than dbg. Or might a narrower first step (debug flags just for some EIC-specific packages in xl) make more sense as a starting point?

[SebastianPaucar]

By the way, I keep seeing a GPU runner failure for the eicweb/eic_container check, unrelated to my changes. Is this a known issue? Could I do something to fix it? @wdconinc

[wdconinc]

Since xl currently builds with no debug flags, adding --debug-source --debug-symbols to xl's installation stage wouldn't capture anything meaningful. It'd need xl to build with debug flags. The cost is that xl has much more specs than dbg. Or might a narrower first step (debug flags just for some EIC-specific packages in xl) make more sense as a starting point?

Yes, right now we are building all (CMake) packages with type Release (see spack-environment/package.yaml: package:all:prefer:build_type=Release). Rather than overriding this entirely, let's start with enabling this just for ROOT and Geant4, and what sits on top of that.

By the way, I keep seeing a GPU runner failure for the eicweb/eic_container check, unrelated to my changes. Is this a known issue? Could I do something to fix it?

This started happening with the upgrade to spack v1.2. To prevent frequent downloads of source archives and buildcache OCI layers we cache the blobs in /var/cache/spack during the container build in a cache volume on the docker builder. This has started to be a lot more flaky for reasons I have not had a chance to dig into. See e.g. https://eicweb.phy.anl.gov/containers/eic_container/-/jobs/8439419#L1475:

   FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/spack/_source-cache/archive/44/.tmp.e871d704a99c.44ada253b1935d34b6801222232d50731fe7c5e3cbcfab47734c85031cfbe4d3.tar.gz'

I think what may have changed is the added e871d704a99c which may be specific to the build, so it essentially bypasses our entire source cache reuse strategy.

[wdconinc]

Rebased to pick up #410.

[SebastianPaucar]

Since xl currently builds with no debug flags, adding --debug-source --debug-symbols to xl's installation stage wouldn't capture anything meaningful. It'd need xl to build with debug flags. The cost is that xl has much more specs than dbg. Or might a narrower first step (debug flags just for some EIC-specific packages in xl) make more sense as a starting point?

Yes, right now we are building all (CMake) packages with type Release (see spack-environment/package.yaml: package:all:prefer:build_type=Release). Rather than overriding this entirely, let's start with enabling this just for ROOT and Geant4, and what sits on top of that.

By the way, I keep seeing a GPU runner failure for the eicweb/eic_container check, unrelated to my changes. Is this a known issue? Could I do something to fix it?

This started happening with the upgrade to spack v1.2. To prevent frequent downloads of source archives and buildcache OCI layers we cache the blobs in /var/cache/spack during the container build in a cache volume on the docker builder. This has started to be a lot more flaky for reasons I have not had a chance to dig into. See e.g. https://eicweb.phy.anl.gov/containers/eic_container/-/jobs/8439419#L1475:

   FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/spack/_source-cache/archive/44/.tmp.e871d704a99c.44ada253b1935d34b6801222232d50731fe7c5e3cbcfab47734c85031cfbe4d3.tar.gz'

I think what may have changed is the added e871d704a99c which may be specific to the build, so it essentially bypasses our entire source cache reuse strategy.

Moved the debug configuration into xl for ROOT, Geant4, and their dependents in xl. Let's see how the CI goes.

On another note, I'm also digging into the /var/cache/spack issue in PR #413

[SebastianPaucar]

Hi @wdconinc, I just added a doc breaking down the debuggable-installations framework in docs/debuggable-installations.md and how it wires into the EIC containers infrastructure, and its current scope. Please check the full flowchart here

On another note, CI checks for this PR are almost done, build eic on amd64_v3 is the only one left; amd64 and arm64 passed successfully. Check the run out here

Is there any way to test this framework in a production case?

[wdconinc]

Is there any way to test this framework in a production case?

There might be. We just last Friday added a new segfault findings platform for production jobs, see https://epic-devcloud.org/prod/panda/segfaults/ (you may need to log in with github, and membership in eic org will give you access). Some of them have been resolved (f-1, f-2, f-3) but my not have rolled out yet. f-4 seems like a promising one to try to reproduce (but in my attempt to reproduce this I got the f-3 failure...).