From 40d564c010fe122711e5cd7a9cdec84d0fd67b80 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 23:24:46 +0000 Subject: [PATCH 1/2] ci: guard the `*-libraries` merge against native-artifact path collisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #388 fixed the corrupt macOS dylib by renaming the three macOS arm64 artifacts out of the `*-libraries` glob, so only `macos-15-metal` ships and it is downloaded by name. That fixes the instance; the glob itself is unchanged, so a future job that ends its artifact name in `-libraries` and writes an already-claimed {OS}/{ARCH} reopens the same hole — silently, because an artifact name says nothing about the path the job wrote. The `package`, `publish-snapshot` and `publish-release` jobs now download the glob UNMERGED (one subdirectory per artifact) and merge it with the new .github/merge-native-artifacts.sh, which fails the job when any relative path is claimed by more than one artifact, and when the glob matched nothing at all (a silently native-library-free default JAR). The check has to run before the merge: a collision leaves exactly one file on the path, so a post-merge assertion such as "exactly one dylib per {OS}/{ARCH}" cannot observe it — the merged tree looks correct and holds a hybrid binary. Verified against a staging tree reproducing the pre-#388 state (three macOS artifacts on Mac/aarch64/libjllama.dylib): fails with the colliding path and its claimants listed; the post-#388 state passes. Docs: CLAUDE.md gains "macOS arm64: three build jobs, one shipped dylib" (the three jobs, which one ships and why, the at-most-one-artifact-per-{OS}/{ARCH} invariant, the guard). TODO.md records the deeper gap this exposed: the macOS Java test jobs each test their own build job's dylib, so nothing ever loads the packaged one — Linux and Windows have smoke-fatjar-* downstream of `package`, macOS has no equivalent, which is why a SIGKILL-on-load binary shipped through three releases with a green pipeline. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Tzo7Yi8SXP6WxhqZXXbsco --- .github/merge-native-artifacts.sh | 93 +++++++++++++++++++++++++++++++ .github/workflows/publish.yml | 42 ++++++++++++-- CLAUDE.md | 43 ++++++++++++++ TODO.md | 26 +++++++++ 4 files changed, 198 insertions(+), 6 deletions(-) create mode 100755 .github/merge-native-artifacts.sh diff --git a/.github/merge-native-artifacts.sh b/.github/merge-native-artifacts.sh new file mode 100755 index 00000000..eb071dc5 --- /dev/null +++ b/.github/merge-native-artifacts.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Merges the per-artifact native-library trees downloaded by the `*-libraries` glob into the +# single default-JAR resource tree — and FAILS LOUD if two artifacts claim the same file path. +# +# Why this exists: the `package` / `publish-snapshot` / `publish-release` jobs pull every build +# job whose artifact name ends in `-libraries` with one globbed `actions/download-artifact`. +# That is convenient (a new CPU platform ships in the default JAR by naming its artifact +# `-libraries`, no packaging change) but it is silently unsafe: an artifact name says +# nothing about which `{OS}/{ARCH}` subdirectory the job's CMake run actually wrote. When two +# artifacts carry the same relative path, `merge-multiple: true` extracts both onto that one +# path and the survivor can be a byte-level hybrid of the two, not either input. +# +# That is exactly what happened to macOS arm64: all three macOS build jobs write +# `Mac/aarch64/libjllama.dylib` (none of them passes -DOS_NAME/-DOS_ARCH, so CMakeLists +# auto-detects the same subdir) and all three used to upload under a `*-libraries` name. The +# published dylib became a hybrid whose ad-hoc linker signature no longer matched its own +# __TEXT pages, so macOS SIGKILLed every process that loaded it — shipped broken in 5.0.6 and +# several 5.0.7 snapshots. The immediate fix renamed those artifacts out of the glob; this +# script is the backstop that stops the same hole from being reopened by a future job. +# +# NOTE ON WHAT *CANNOT* WORK AS A GUARD: asserting "exactly one library per {OS}/{ARCH}" on the +# merged tree does not detect this. The collision overwrites one path, so the merged tree still +# holds exactly one file there — a corrupt one. The collision is only observable BEFORE the +# merge, which is why this script does the merge itself instead of checking afterwards. +# +# Usage: merge-native-artifacts.sh +# output of `actions/download-artifact` with `pattern: "*-libraries"` and +# `merge-multiple: false`, i.e. one subdirectory per artifact name. +# the tree the artifacts are merged into, e.g. +# llama/src/main/resources/net/ladenthin/llama/ +# +# Fail-loud: aborts when the staging directory holds no artifacts (a silently empty default JAR +# is worse than a red job) and when any relative path is claimed by more than one artifact. + +set -euo pipefail + +STAGING="${1:?usage: merge-native-artifacts.sh }" +DEST="${2:?usage: merge-native-artifacts.sh }" + +if [ ! -d "$STAGING" ]; then + echo "::error::staging directory '$STAGING' does not exist — the globbed download did not run." >&2 + exit 1 +fi + +# One subdirectory per downloaded artifact. Depth 1 only: everything below is artifact content. +artifacts=() +while IFS= read -r d; do artifacts+=("$(basename "$d")"); done < <(find "$STAGING" -mindepth 1 -maxdepth 1 -type d | sort) + +if [ "${#artifacts[@]}" -eq 0 ]; then + echo "::error::no '*-libraries' artifacts found in '$STAGING' — the default JAR would ship without native libraries." >&2 + exit 1 +fi + +echo "Merging ${#artifacts[@]} native-library artifact(s) into $DEST" +for a in "${artifacts[@]}"; do echo " - $a"; done + +# relpath -> space-separated list of artifacts that carry it. Bash 3.2 (macOS) has no +# associative arrays, so this stays a sorted "\t" stream processed by awk. +collisions="$( + for a in "${artifacts[@]}"; do + (cd "$STAGING/$a" && find . -type f | sed 's|^\./||' | while IFS= read -r f; do printf '%s\t%s\n' "$f" "$a"; done) + done | sort | awk -F'\t' ' + { if ($1 == prev) { owners = owners " " $2; n++ } else { if (n > 1) print prev "\t" owners; prev = $1; owners = $2; n = 1 } } + END { if (n > 1) print prev "\t" owners } + ' +)" + +if [ -n "$collisions" ]; then + echo "::error::two or more '*-libraries' artifacts write the same path — merging them would produce a hybrid, corrupt native library." >&2 + while IFS=$'\t' read -r path owners; do + echo "::error:: $path <- claimed by:$owners" >&2 + done <<< "$collisions" + cat >&2 <<'EOF' +::error::Fix: only ONE artifact per {OS}/{ARCH} may be named `*-libraries`. Rename the extra +::error::build jobs' artifacts outside the glob (as the macOS jobs do: macos-15-metal / +::error::macos-14-metal / macos-15-no-metal) and download the variant that ships explicitly by +::error::name. See CLAUDE.md, "macOS arm64: three build jobs, one shipped dylib". +EOF + exit 1 +fi + +mkdir -p "$DEST" +for a in "${artifacts[@]}"; do + cp -R "$STAGING/$a/." "$DEST/" +done + +echo "Merged native tree:" +find "$DEST" -type f | sort | sed 's|^| |' diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e9c6527f..bfa107df 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2747,11 +2747,21 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 + # Downloaded UNMERGED (one subdirectory per artifact name) on purpose: `merge-multiple: true` + # would silently overwrite — and can byte-level interleave — two artifacts that carry the same + # relative path, which is how the corrupt macOS dylib shipped. merge-native-artifacts.sh does + # the merge instead, and fails the job if any {OS}/{ARCH} path is claimed by more than one + # `*-libraries` artifact. A post-merge check cannot catch this (the collision leaves exactly + # one file on the path — a corrupt one), so the check has to happen before the merge. - uses: actions/download-artifact@v8 with: pattern: "*-libraries" - merge-multiple: true - path: ${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/ + path: ${{ github.workspace }}/native-artifacts/ + - name: Merge native libraries into the default resource tree (collision-checked) + run: | + bash .github/merge-native-artifacts.sh \ + "${{ github.workspace }}/native-artifacts" \ + "${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/" # All three macOS arm64 build jobs emit the same path Mac/aarch64/libjllama.dylib, so their # artifact names are kept outside the "*-libraries" glob above — merging them would drop # three different dylibs onto one file. The variant that ships is selected explicitly: @@ -3042,11 +3052,21 @@ jobs: contents: write steps: - uses: actions/checkout@v7 + # Downloaded UNMERGED (one subdirectory per artifact name) on purpose: `merge-multiple: true` + # would silently overwrite — and can byte-level interleave — two artifacts that carry the same + # relative path, which is how the corrupt macOS dylib shipped. merge-native-artifacts.sh does + # the merge instead, and fails the job if any {OS}/{ARCH} path is claimed by more than one + # `*-libraries` artifact. A post-merge check cannot catch this (the collision leaves exactly + # one file on the path — a corrupt one), so the check has to happen before the merge. - uses: actions/download-artifact@v8 with: pattern: "*-libraries" - merge-multiple: true - path: ${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/ + path: ${{ github.workspace }}/native-artifacts/ + - name: Merge native libraries into the default resource tree (collision-checked) + run: | + bash .github/merge-native-artifacts.sh \ + "${{ github.workspace }}/native-artifacts" \ + "${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/" # All three macOS arm64 build jobs emit the same path Mac/aarch64/libjllama.dylib, so their # artifact names are kept outside the "*-libraries" glob above — merging them would drop # three different dylibs onto one file. The variant that ships is selected explicitly: @@ -3264,11 +3284,21 @@ jobs: contents: write steps: - uses: actions/checkout@v7 + # Downloaded UNMERGED (one subdirectory per artifact name) on purpose: `merge-multiple: true` + # would silently overwrite — and can byte-level interleave — two artifacts that carry the same + # relative path, which is how the corrupt macOS dylib shipped. merge-native-artifacts.sh does + # the merge instead, and fails the job if any {OS}/{ARCH} path is claimed by more than one + # `*-libraries` artifact. A post-merge check cannot catch this (the collision leaves exactly + # one file on the path — a corrupt one), so the check has to happen before the merge. - uses: actions/download-artifact@v8 with: pattern: "*-libraries" - merge-multiple: true - path: ${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/ + path: ${{ github.workspace }}/native-artifacts/ + - name: Merge native libraries into the default resource tree (collision-checked) + run: | + bash .github/merge-native-artifacts.sh \ + "${{ github.workspace }}/native-artifacts" \ + "${{ github.workspace }}/llama/src/main/resources/net/ladenthin/llama/" # All three macOS arm64 build jobs emit the same path Mac/aarch64/libjllama.dylib, so their # artifact names are kept outside the "*-libraries" glob above — merging them would drop # three different dylibs onto one file. The variant that ships is selected explicitly: diff --git a/CLAUDE.md b/CLAUDE.md index 1637c551..93f37af8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -343,6 +343,49 @@ pinned to a specific version): if a URL/version 404s in CI, the job fails loud a `src/main/resources_{linux_rocm,windows_rocm,linux_sycl_fp16,linux_sycl_fp32,windows_sycl,linux_openvino,windows_openvino}/` are all git-ignored (staged by CI, never committed). +## macOS arm64: three build jobs, one shipped dylib + +macOS arm64 is the **only** `{OS}/{ARCH}` built by more than one job, and it has **no classifier** — +all three variants ship (or don't) into the same default-JAR path `Mac/aarch64/libjllama.dylib`: + +| Job | Build flags | Artifact | Role | +|---|---|---|---| +| `build-macos-arm64-metal-15` (macos-15) | `-DLLAMA_METAL_EMBED_LIBRARY=ON -DGGML_NATIVE=OFF` | `macos-15-metal` | **shipped** in the default JAR | +| `build-macos-arm64-metal` (macos-14) | `-DLLAMA_METAL_EMBED_LIBRARY=ON` (host-native) | `macos-14-metal` | test-only | +| `build-macos-arm64-no-metal` (macos-15) | `-DLLAMA_METAL=OFF -DGGML_NATIVE=OFF` | `macos-15-no-metal` | test-only | + +None of the three passes `-DOS_NAME`/`-DOS_ARCH`, so `CMakeLists.txt` auto-detects the **same** +output subdirectory in all three. The shipped variant is `macos-15-metal` because it is the only one +with **both** Metal **and** `GGML_NATIVE=OFF` (portable across Apple-silicon generations); the other +two exist to prove the no-Metal path and the macos-14 SDK still build and pass the Java suite. + +**The invariant: at most one `*-libraries` artifact per `{OS}/{ARCH}`.** The `package`, +`publish-snapshot` and `publish-release` jobs collect the default JAR's natives with one globbed +`actions/download-artifact` (`pattern: "*-libraries"`). An artifact *name* says nothing about which +subdirectory the job actually wrote, so two artifacts sharing a relative path get extracted onto one +file — and the survivor can be a **byte-level hybrid** of both, not either input. All three macOS +jobs used to upload under a `*-libraries` name: the published dylib's ad-hoc linker signature then no +longer matched its own `__TEXT` pages and macOS **SIGKILLed every process that loaded it** (shipped +broken in 5.0.6 and several 5.0.7 snapshots; 66/4078 and 1141/4097 code pages failed their stored +hashes). Windows already avoided this by naming its MSVC variants outside the glob; macOS now does +the same, and the shipped variant is chosen by an **explicit download step by name**, never by which +artifact name happens to match a glob. + +**The guard: `.github/merge-native-artifacts.sh`.** The three consumer jobs download the glob +**unmerged** (`merge-multiple` off → one subdirectory per artifact) and let that script do the merge. +It fails the job when any relative path is claimed by more than one `*-libraries` artifact, and when +the glob matched nothing at all (a silently native-library-free default JAR). Note **why the check +runs before the merge**: a collision still leaves exactly one file on the path, so a post-merge +assertion like "exactly one dylib per `{OS}/{ARCH}`" cannot see it — the collision is only observable +while the artifacts are still separate. A future job that reopens the hole (a second job writing +`Mac/aarch64`, or a new platform whose auto-detected subdir clashes) reds the pipeline instead of +shipping a corrupt binary. + +**Still uncovered (see [`TODO.md`](TODO.md)):** the three macOS Java test jobs each test the dylib +*their own job* built, so nothing exercises the **merged/packaged** artifact on macOS. Linux and +Windows have `smoke-fatjar-*` jobs downstream of `package`; macOS has none — which is why this bug +reached three releases with a fully green pipeline. + ## All-backends server fat jars (GitHub Release assets, never Maven Central) Every pipeline run assembles **per-OS multi-backend server fat jars** and, on the release diff --git a/TODO.md b/TODO.md index 5c3b4681..a2e252ad 100644 --- a/TODO.md +++ b/TODO.md @@ -203,6 +203,32 @@ real arm64 hardware and the Adreno/OpenCL flavor. Treat LLaMAndroid as prior art **Out of scope until evidence supports it**: actually implementing any of the above. This entry exists so that when someone asks "can I ship java-llama.cpp as a single 30 MB binary?" the answer points to a concrete investigation plan rather than restarting from zero. +### macOS has no post-`package` smoke test (the gap that let a corrupt dylib ship) + +The three macOS Java test jobs each run against the dylib **their own build job** produced, so +nothing in the pipeline ever loads the dylib that actually goes **into the published JAR**. That is +why the `*-libraries` glob collision (three macOS artifacts overwriting +`Mac/aarch64/libjllama.dylib` into a hybrid that macOS SIGKILLs on load — see CLAUDE.md, "macOS +arm64: three build jobs, one shipped dylib") shipped in 5.0.6 and several 5.0.7 snapshots with an +all-green pipeline. Linux and Windows do have the equivalent gate: `smoke-fatjar-linux` / +`smoke-fatjar-windows` run `java -jar` against the **packaged** artifact downstream of `package`. + +`.github/merge-native-artifacts.sh` now closes the specific hole (fails the merge when two +`*-libraries` artifacts claim one path), but it is a guard on *one known* corruption mechanism, not +an end-to-end check that the shipped macOS binary loads. + +- **Add a `smoke-fatjar-macos` job** (`needs: [package-fatjars, verify-model-cache]`, `runs-on: + macos-15`), mirroring the two existing smoke jobs: download `llama-fatjars`, run + `.github/smoke-test-fatjar.sh` against the `all-macos-aarch64` / default fat jar. On macOS this + also catches the code-signature class of failure for free, since `dyld` refuses to map a dylib + whose ad-hoc signature does not match its pages. +- **Cheaper interim option** if a full model-backed macOS smoke is too slow: a macOS job downstream + of `package` that only downloads `llama-jars`, extracts the dylib, and runs `codesign --verify + --strict` plus a `NativeLibraryLoadSmokeTest`-style `System.load` (no GGUF needed). That alone + would have caught this bug. + +Not a release gate until it has run flake-free (same rollout the Android emulator job followed). + ## Open — cross-cutting (slice for this repo) - **jqwik pin policy** — see [`../workspace/policies/jqwik-prompt-injection.md`](../workspace/policies/jqwik-prompt-injection.md). `jqwik.version ≤ 1.9.3` is mandatory. From 2662e8868e987b0ee55d85bd6e82d9e689c77ca4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 23:43:09 +0000 Subject: [PATCH 2/2] =?UTF-8?q?ci:=20add=20smoke-fatjar-macos=20=E2=80=94?= =?UTF-8?q?=20verify=20the=20dylib=20inside=20the=20packaged=20jar?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit macOS was the one platform with no post-`package` gate: the three macOS Java test jobs each run against the dylib their own build job produced, so nothing ever loaded the copy that goes into the published jar. That is why a hybrid libjllama.dylib — whose ad-hoc signature no longer covered its own __TEXT pages, so macOS SIGKILLs any process that loads it — shipped in 5.0.6 and several 5.0.7 snapshots with an all-green pipeline. Linux and Windows already had smoke-fatjar-linux / -windows downstream of `package`. The new job downloads llama-jars and runs .github/smoke-native-macos.sh: extract Mac/aarch64/libjllama.dylib from the default fat jar, `codesign --verify --strict` it (this re-hashes the code pages against the signature's stored hashes, so a dylib assembled from two builds fails exactly here), then load it in a real JVM via the JDK single-file source launcher (.github/smoke/NativeLoadSmoke.java) and cross JNI for getLlamaCppBuildInfo(), checked against the LlamaCppVersion pin. It targets the default fat jar because there is no all-macos-* fat jar to target — macOS has no GPU classifier, Metal ships in the default jar, so package-fatjars builds no macOS variant. It asserts native loadability rather than a CLI exit code, so it cannot reuse smoke-test-fatjar.sh, whose backend-manifest grep never matches the manifest-less default jar. Model-free by design (~1 min, no GGUF, no cache restore, no network): a slow smoke gets made non-gating and then the gap reopens. This is the macOS member of the cross-repo "no release asset is attached that CI has not run" convention; BitcoinAddressFinder and srcmorph implement it with a shared smoke-fatjar-cli.sh. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Tzo7Yi8SXP6WxhqZXXbsco --- .github/smoke-native-macos.sh | 77 ++++++++++++++++++++++++++++++ .github/smoke/NativeLoadSmoke.java | 45 +++++++++++++++++ .github/workflows/publish.yml | 38 ++++++++++++++- CLAUDE.md | 26 ++++++++-- TODO.md | 43 +++++++---------- 5 files changed, 198 insertions(+), 31 deletions(-) create mode 100755 .github/smoke-native-macos.sh create mode 100644 .github/smoke/NativeLoadSmoke.java diff --git a/.github/smoke-native-macos.sh b/.github/smoke-native-macos.sh new file mode 100755 index 00000000..a8e2902b --- /dev/null +++ b/.github/smoke-native-macos.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# macOS post-`package` smoke: verifies the libjllama.dylib that is actually INSIDE the packaged jar +# — its code signature, and that a JVM can load it and cross the JNI boundary. +# +# Why this exists (the gap it closes): the three macOS Java test jobs each run against the dylib +# THEIR OWN build job produced. Nothing in the pipeline ever loaded the one that goes into the +# published jar. So when the `*-libraries` artifact glob merged three different macOS dylibs onto +# one path and produced a byte-level hybrid, the result — a library whose ad-hoc linker signature no +# longer matched its own __TEXT pages, which macOS SIGKILLs on load — shipped in 5.0.6 and several +# 5.0.7 snapshots with an all-green pipeline. Linux and Windows already had the equivalent gate +# (`smoke-fatjar-linux` / `smoke-fatjar-windows`, downstream of `package`); macOS had none. +# +# This is the macOS member of the cross-repo "no release asset is attached that CI has not run" +# convention (workspace/policies/fat-jar-release-assets.md). It is NOT the shared +# smoke-fatjar-cli.sh that BitcoinAddressFinder and srcmorph run: this jar's Main-Class is a server +# that never exits, and the assertion that matters here is native-library loadability, not a CLI +# exit code. Same job shape, repo-specific assertions. +# +# Deliberately model-free: no GGUF, no cache restore, no network — it runs in ~1 min. A full +# model-backed macOS server smoke would be strictly more, but the failure class that actually +# shipped is caught here, so this is the version that is cheap enough to always run. +# +# Usage: smoke-native-macos.sh +# directory to search for the jar (recursively) +# filename glob; must match EXACTLY ONE jar + +set -euo pipefail + +JAR_DIR="${1:?usage: smoke-native-macos.sh }" +JAR_GLOB="${2:?usage: smoke-native-macos.sh }" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +fail() { + echo "::error::$*" >&2 + exit 1 +} + +[ -d "$JAR_DIR" ] || fail "jar directory '$JAR_DIR' does not exist" + +jars=() +while IFS= read -r j; do jars+=("$j"); done < <(find "$JAR_DIR" -type f -name "$JAR_GLOB" | sort) +[ "${#jars[@]}" -eq 1 ] \ + || fail "expected exactly 1 jar matching '$JAR_GLOB' under '$JAR_DIR', got ${#jars[@]}: ${jars[*]:-none}" +JAR="$(cd "$(dirname "${jars[0]}")" && pwd)/$(basename "${jars[0]}")" +echo "smoke jar: $JAR" + +DYLIB_ENTRY="net/ladenthin/llama/Mac/aarch64/libjllama.dylib" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +unzip -o -q "$JAR" "$DYLIB_ENTRY" -d "$WORK" \ + || fail "the jar does not contain $DYLIB_ENTRY — the macOS natives never reached the package job" +DYLIB="$WORK/$DYLIB_ENTRY" +echo "extracted: $(cd "$(dirname "$DYLIB")" && pwd)/$(basename "$DYLIB") ($(wc -c < "$DYLIB") bytes)" + +# 1) Signature vs. content. `--strict` re-hashes the code pages and compares them against the +# signature's stored hashes, so a dylib assembled from two different builds fails here with the +# exact page mismatch — the direct check for the shipped corruption. An ad-hoc signature (what +# the linker emits on arm64) is expected and fine; only a MISMATCH is a failure. +echo "== codesign --verify --strict ==" +codesign --verify --strict --verbose=2 "$DYLIB" \ + || fail "code signature does not match the dylib's own pages — the packaged library is corrupt (macOS would SIGKILL any process that loads it)" + +# 2) The JVM must actually be able to map it and call through JNI. This is what a consumer does, +# and it is the only check that covers load-time failures the signature check cannot see +# (missing dependent library, wrong architecture, unresolved JNI_OnLoad class lookup). +echo "== JVM load + JNI round-trip ==" +java -cp "$JAR" "$SCRIPT_DIR/smoke/NativeLoadSmoke.java" \ + || fail "the packaged native library did not load in a JVM" + +echo "smoke test PASSED" diff --git a/.github/smoke/NativeLoadSmoke.java b/.github/smoke/NativeLoadSmoke.java new file mode 100644 index 00000000..b6a8717d --- /dev/null +++ b/.github/smoke/NativeLoadSmoke.java @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: 2026 Bernard Ladenthin +// +// SPDX-License-Identifier: MIT OR Apache-2.0 + +import net.ladenthin.llama.LlamaModel; +import net.ladenthin.llama.value.LlamaCppVersion; + +/** + * Model-free load smoke for the PACKAGED native library, run by {@code .github/smoke-native-macos.sh} + * via the JDK single-file source launcher ({@code java -cp NativeLoadSmoke.java}) — no + * Maven, no test framework, no GGUF. + * + *

Touching {@link LlamaModel} runs its static initializer, which is + * {@code LlamaLoader.initialize() -> System.load() -> JNI_OnLoad}: the library is extracted from the + * jar, mapped by the OS loader, and every class its {@code FindClass} calls reference is resolved. + * On macOS that mapping is also where a dylib whose ad-hoc signature no longer covers its own + * {@code __TEXT} pages is rejected — the failure that shipped in 5.0.6. Then + * {@link LlamaModel#getLlamaCppBuildInfo()} crosses the JNI boundary for real and is checked against + * the compile-time pin, so a jar carrying a stale or foreign library fails here rather than in a + * user's process.

+ * + *

This is the packaged-artifact counterpart of {@code NativeLibraryLoadSmokeTest}, which asserts + * the same two things against {@code target/classes} during {@code mvn test} — that one cannot see + * what the assembled jar contains.

+ */ +public final class NativeLoadSmoke { + + private NativeLoadSmoke() {} + + public static void main(String[] args) { + final String pinned = LlamaCppVersion.LLAMA_CPP_VERSION; + + // Forces LlamaLoader.initialize() -> System.load() -> JNI_OnLoad. + final String build = LlamaModel.getLlamaCppBuildInfo(); + + if (build == null || build.isEmpty()) { + throw new IllegalStateException("getLlamaCppBuildInfo() returned no build identifier"); + } + if (!build.startsWith(pinned)) { + throw new IllegalStateException("packaged native library reports llama.cpp build '" + build + + "' but this jar pins '" + pinned + "' — the jar carries the wrong native library"); + } + System.out.println("native load smoke OK: pinned=" + pinned + " linked=" + build); + } +} diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index bfa107df..df486cf3 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2994,6 +2994,40 @@ jobs: server-err.log if-no-files-found: warn + # --------------------------------------------------------------------------- + # macOS member of the cross-repo "no release asset is attached that CI has not run" convention + # (workspace/policies/fat-jar-release-assets.md; BitcoinAddressFinder and srcmorph run the shared + # .github/smoke-fatjar-cli.sh in the same job shape). It closes the gap that let a corrupt dylib + # ship: the three macOS Java test jobs each test the dylib THEIR OWN build job produced, so until + # now nothing ever loaded the one that goes into the published jar — see CLAUDE.md, "macOS arm64: + # three build jobs, one shipped dylib". + # + # macOS-specific in two ways. It targets the DEFAULT fat jar from `llama-jars`, because there is + # no `all-macos-*` fat jar to target: macOS has no GPU classifier (Metal ships in the default + # jar), so package-fatjars builds no macOS variant. And it asserts native loadability rather than + # a CLI exit code — this jar's Main-Class is a server that never returns, and `codesign --strict` + # is what actually detects a dylib assembled from two builds. Deliberately model-free (no GGUF, + # no cache restore, ~1 min): a full model-backed macOS server smoke would be strictly more, but + # this catches the failure class that shipped and is cheap enough to always run. + # --------------------------------------------------------------------------- + + smoke-fatjar-macos: + name: Smoke test packaged natives (macOS) + needs: [package] + runs-on: macos-15 + steps: + - uses: actions/checkout@v7 + - uses: actions/download-artifact@v8 + with: + name: llama-jars + path: jars/ + - uses: actions/setup-java@v5 + with: + distribution: 'temurin' + java-version: ${{ env.JAVA_VERSION }} + - name: Run packaged-native smoke test + run: .github/smoke-native-macos.sh jars 'llama-*-jar-with-dependencies.jar' + report: name: Report needs: [package] @@ -3044,7 +3078,7 @@ jobs: publish-snapshot: name: Publish Snapshot to Central - needs: [check-snapshot, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows] + needs: [check-snapshot, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-macos] if: needs.check-snapshot.result == 'success' && inputs.publish_to_central runs-on: ubuntu-latest environment: maven-central @@ -3277,7 +3311,7 @@ jobs: publish-release: name: Publish Release to Central if: needs.check-tag.result == 'success' && inputs.publish_to_central - needs: [check-tag, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows] + needs: [check-tag, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-macos] runs-on: ubuntu-latest environment: maven-central permissions: diff --git a/CLAUDE.md b/CLAUDE.md index 93f37af8..d19d11ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -381,10 +381,28 @@ while the artifacts are still separate. A future job that reopens the hole (a se `Mac/aarch64`, or a new platform whose auto-detected subdir clashes) reds the pipeline instead of shipping a corrupt binary. -**Still uncovered (see [`TODO.md`](TODO.md)):** the three macOS Java test jobs each test the dylib -*their own job* built, so nothing exercises the **merged/packaged** artifact on macOS. Linux and -Windows have `smoke-fatjar-*` jobs downstream of `package`; macOS has none — which is why this bug -reached three releases with a fully green pipeline. +**The end-to-end gate: `smoke-fatjar-macos`.** The three macOS Java test jobs each test the dylib +*their own job* built, so until now nothing exercised the **packaged** artifact on macOS — Linux and +Windows had `smoke-fatjar-*` downstream of `package`, macOS had none, which is why this bug reached +three releases with a fully green pipeline. The job (`needs: [package]`, `macos-15`, gates both +publish jobs) downloads `llama-jars` and runs `.github/smoke-native-macos.sh`, which extracts +`Mac/aarch64/libjllama.dylib` from the **default fat jar** and asserts two things: `codesign --verify +--strict` (re-hashes the code pages against the signature's stored hashes — the direct check for a +dylib assembled from two builds) and a real JVM load via `.github/smoke/NativeLoadSmoke.java` +(`java -cp …`, the JDK single-file source launcher), which forces +`LlamaLoader.initialize() → System.load() → JNI_OnLoad` and then crosses JNI for +`getLlamaCppBuildInfo()`, checked against the `LlamaCppVersion` pin. + +Two macOS specifics: it targets the **default** fat jar because there is no `all-macos-*` fat jar to +target (macOS has no GPU classifier — Metal is in the default jar — so `package-fatjars` builds no +macOS variant), and it asserts native loadability rather than a CLI exit code, so it cannot reuse +`smoke-test-fatjar.sh` (whose backend-manifest grep never matches the manifest-less default jar). +Deliberately model-free (~1 min, no GGUF/cache restore/network): a full model-backed macOS server +smoke would be strictly more, but this catches the failure class that actually shipped and is cheap +enough to always run. It is the macOS member of the cross-repo convention in +[`../workspace/policies/fat-jar-release-assets.md`](../workspace/policies/fat-jar-release-assets.md) +("No release asset is attached that CI has not run"), which BAF and srcmorph implement with a shared +`smoke-fatjar-cli.sh`. ## All-backends server fat jars (GitHub Release assets, never Maven Central) diff --git a/TODO.md b/TODO.md index a2e252ad..22893e42 100644 --- a/TODO.md +++ b/TODO.md @@ -203,31 +203,24 @@ real arm64 hardware and the Adreno/OpenCL flavor. Treat LLaMAndroid as prior art **Out of scope until evidence supports it**: actually implementing any of the above. This entry exists so that when someone asks "can I ship java-llama.cpp as a single 30 MB binary?" the answer points to a concrete investigation plan rather than restarting from zero. -### macOS has no post-`package` smoke test (the gap that let a corrupt dylib ship) - -The three macOS Java test jobs each run against the dylib **their own build job** produced, so -nothing in the pipeline ever loads the dylib that actually goes **into the published JAR**. That is -why the `*-libraries` glob collision (three macOS artifacts overwriting -`Mac/aarch64/libjllama.dylib` into a hybrid that macOS SIGKILLs on load — see CLAUDE.md, "macOS -arm64: three build jobs, one shipped dylib") shipped in 5.0.6 and several 5.0.7 snapshots with an -all-green pipeline. Linux and Windows do have the equivalent gate: `smoke-fatjar-linux` / -`smoke-fatjar-windows` run `java -jar` against the **packaged** artifact downstream of `package`. - -`.github/merge-native-artifacts.sh` now closes the specific hole (fails the merge when two -`*-libraries` artifacts claim one path), but it is a guard on *one known* corruption mechanism, not -an end-to-end check that the shipped macOS binary loads. - -- **Add a `smoke-fatjar-macos` job** (`needs: [package-fatjars, verify-model-cache]`, `runs-on: - macos-15`), mirroring the two existing smoke jobs: download `llama-fatjars`, run - `.github/smoke-test-fatjar.sh` against the `all-macos-aarch64` / default fat jar. On macOS this - also catches the code-signature class of failure for free, since `dyld` refuses to map a dylib - whose ad-hoc signature does not match its pages. -- **Cheaper interim option** if a full model-backed macOS smoke is too slow: a macOS job downstream - of `package` that only downloads `llama-jars`, extracts the dylib, and runs `codesign --verify - --strict` plus a `NativeLibraryLoadSmokeTest`-style `System.load` (no GGUF needed). That alone - would have caught this bug. - -Not a release gate until it has run flake-free (same rollout the Android emulator job followed). +### macOS packaged-artifact gate — landed cheap, optional depth remains + +**Done:** `smoke-fatjar-macos` (`needs: [package]`, gates both publish jobs) now verifies the dylib +inside the packaged jar — `codesign --verify --strict` plus a real JVM load and JNI round-trip via +`.github/smoke/NativeLoadSmoke.java`. See CLAUDE.md, "macOS arm64: three build jobs, one shipped +dylib". This closes the gap that let a SIGKILL-on-load binary ship through three releases with a +green pipeline, and is the macOS member of the cross-repo convention in +[`../workspace/policies/fat-jar-release-assets.md`](../workspace/policies/fat-jar-release-assets.md). + +**Optional depth, not scheduled:** a full model-backed macOS server smoke (poll `/health`, assert a +`/v1/chat/completions` choice) as Linux and Windows run. It would need `verify-model-cache` + +a cache restore, and — since there is no `all-macos-*` fat jar — either a macOS variant from +`package-fatjars` or a `smoke-test-fatjar.sh` flag making the backend-manifest grep optional for the +manifest-less default jar. Worth doing only if a macOS-specific *inference* regression ever appears; +the load-time failure class is already covered, and a slow smoke tends to get made non-gating. + +**Not yet observed green in CI** — the job and the two sibling-repo smokes landed in one change set +and have only run locally so far. ## Open — cross-cutting (slice for this repo)