Add caching for dependencies and pip to CI

[johvincau]

Summary

Test suite dependencies can be cached to shorten install times. This PR targets both built dependencies (njoy, moab + dagmc, libmesh) as well as all unique configurations of Python called by the job matrix, so no time is spent either compiling dependencies or building new wheels on each runner.

This PR resolves #2140.

When all relevant caches can be restored, this feature brings the most expensive job install times in line with pure OpenMC jobs while providing modest install time improvements to all others.

The baseline install times come from a merge from openmc/develop into my fork, so there was no caching beyond the xs cache. The cached install times come from a test run on my branch. The time to restore from cache is around several seconds, which is not included in the above data.

Dependencies

For dependencies built from source, gha-cache-keys.sh checks the following tags for the corresponding commit hash, which is then used as the cache key.

• NJOY = 2016.78
• MOAB = 5.5.1
• LIBMESH = v1.7.1
• DAGMC has no specified version; the key references HEAD on its develop branch.

Each one of these dependencies has its own separate cache, and in the case of libmesh, two caches for both an mpi and non-mpi build. A new binary will be built and cached if the commit hash changes.

Python Caching

The ~/.cache/pip directory is cached with two hashes in its key. The first is the hash of requirements-mpi-X.txt, where X is either 'y' or 'n' based on the job. These files are used purely to generate the hash, and not as a reference for installing dependencies on the runner (this has been handled by . The second is the hash of mpi-deps.txt which records the versions of mpi apt dependencies dpkg -W libmpich-dev libhdf5-mpich-dev. This file, also generated by gha-cache-keys.sh, is empty if the job does not need mpi, so the resulting hash is just the hash of nothing. The second hash is included so that in the event of an update to the mpi apt dependencies, a new pip cache built against it will be stored. The pip dependencies for MPI installs, mpi4py and h5py, should be built against these dependencies.

Overall, 5 separate configurations of Python are stored:
Python 3.12 with and without MPI, 3.13, 3.14, and 3.14t, at around 250 MB each.

Checklist

• I have performed a self-review of my own code
• I have run clang-format (version 18) on any C++ source files (if applicable)
• I have followed the style guidelines for Python source files (if applicable)
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works (if applicable)

[shimwell]

A different path is used when restoring NJOY

[shimwell]

I think the restore NJOY and dir used to build NJOY are different, so where the tests run we would skip NJOY related tests due to this pytest setting

openmc/tests/unit_tests/__init__.py

Lines 8 to 9 in 03b036a

	needs_njoy = pytest.mark.skipif(shutil.which('njoy') is None,
	reason="NJOY not installed")

perhaps we should preserve the two artifcats the cache path declaire, e.g. replace the blanket rm -rf with a prune that keeps njoy and libnjoy.so, and switch the guard to the chace action's own output.

[jon-proximafusion]

I think this guard might break the dagmc job. MOAB's key looks like it basically never changes, whereas the DAGMC key follows HEAD of svalinn/dagmc develop, so I'd expect a lot of runs to hit on MOAB and miss on DAGMC. If that happens then this test passes, gha-install-dagmc.sh runs, and its mkdir MOAB might die with File exists because of set -e. CI could then go red next time dagmc gets a commit.

Would it be worth using the cache-hit output from each cache step rather than checking whether the directory exists? I think that would also mean you don't need metaconfigure in the cache paths just to make this check work.

[jon-proximafusion]

I don't think this is picking up the tag you want. --tags doesn't seem to filter out other matches:

$ git ls-remote https://bitbucket.org/fathomteam/moab.git --tags Version5.1.0
7bde9dfb84a8cbeb5a748f43150699d9988df6be	refs/heads/Version5.1.0

That looks like a branch head for 5.1.0, whereas gha-install-dagmc.sh builds tag 5.5.1. If that's right, bumping MOAB_BRANCH wouldn't change the key, so we might carry on using the old MOAB, and a push to the 5.1.0 branch would bust the cache for nothing.

Line 4 might have something similar going on. With both HEAD and --tags 2016.78 you get two lines back and head -c 15 grabs the first, so I think the key ends up as ac5adf5f33d893e (their HEAD) rather than 71a76bc from the tag. If so, NJOY would get rebuilt whenever their default branch moves.

njoy, moab and libmesh all look pinned to fixed tags anyway, so would it be simpler to hardcode the version in the key and skip ls-remote for those three? DAGMC seems like the only one that needs it.

[jon-proximafusion]

Separate from the njoy thing: as far as I can tell the keys don't include a hash of the script that builds each dependency. If that's right, then changing a cmake or configure flag in gha-install-*.sh wouldn't change the key, the old binary would come back, and the ! -d check would skip the rebuild. I don't think it would recover on its own either, since a primary key hit doesn't re-save.

This might have already happened here. Your fork pulled down 1.2 MB for NJOY2016-ac5adf5f33d893e, from before you added the rm -rf, while the run here saved 14 KB under the same key:

22:58:01  + rm -rf /home/runner/NJOY2016/build
23:16:56  Sent 14096 of 14096 (100.0%)
23:17:00  Cache saved with key: NJOY2016-ac5adf5f33d893e

If that's what's going on, the install times in the description would have come off a cache built by older versions of these scripts. Every dependency cache looks like it missed in the run here, so I'm not sure the restore path has really been exercised in this repo yet. Would you mind trying it on a throwaway branch, two runs back to back, and checking that njoy is on PATH and the njoy tests run rather than skip? I think you'd also need to delete the existing NJOY2016-ac5adf5f33d893e cache on develop and on your fork first, otherwise the 14 KB one would stick around.