diff --git a/CLAUDE.md b/CLAUDE.md index 06c74083..6cbf976f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,8 +14,8 @@ Current CUDA version: **13.2** To change the CUDA version, update the following **three** places: -1. **`.github/build_cuda_linux.sh`** — Line 10: `sudo dnf install -y cuda-toolkit-13-2` -2. **`.github/build_cuda_linux.sh`** — Line 12: `-DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc` +1. **`.github/build_cuda_linux.sh`** — Line 16: `sudo dnf install -y cuda-toolkit-13-2` +2. **`.github/build_cuda_linux.sh`** — Line 41: `-DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc` 3. **`llama/pom.xml`** — The `` tag in the `cuda` jar execution: `cuda13-linux-x86-64` Also update the header comment in `build_cuda_linux.sh` and the job name in `.github/workflows/release.yaml` for clarity. @@ -557,15 +557,15 @@ are kept jllama-only) is explained in the cross-repo status under "Deliberate no The fetched llama.cpp source is patched before it compiles, via a generic mechanism: -- **`patches/`** (repo root) — drop any number of `*.patch` / `*.diff` files here. They are applied +- **`llama/patches/`** — drop any number of `*.patch` / `*.diff` files here. They are applied in **filename order** (use a numeric prefix, e.g. `0001-`, `0002-`), so keep them independent or ordered. Each must be a `git apply`-compatible unified diff with paths relative to the llama.cpp source root (`a/common/arg.cpp` / `b/common/arg.cpp`, i.e. `-p1`). -- **`cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on +- **`llama/cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on Linux/macOS/Windows), **idempotent** (`git apply --reverse --check` skips already-applied patches so a reconfigure never double-applies) and **fail-loud** (a patch that no longer applies aborts the configure — a stale patch can't be silently dropped from a release build). -- **`CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it +- **`llama/CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it runs for **every** C++ build (all CI jobs *and* local `cmake -B build`) from one place — no per-build-step plumbing. @@ -604,7 +604,7 @@ prefill behavior shows up, re-check `tools/server/server-context.cpp`'s `create_ (`test_reasoning_budget_tokens_per_request` / `test_reasoning_budget_message_per_request`, byte-identical body). No local patch needed — the tree already matches what `0004` used to add. -## OuteTTS build-time extraction (`cmake/generate-tts-upstream.cmake`) +## OuteTTS build-time extraction (`llama/cmake/generate-tts-upstream.cmake`) The `TextToSpeech` native pipeline reuses llama.cpp's OuteTTS helpers (`tools/tts/tts.cpp`) **without hand-copying them**. A verbatim copy would be a DRY/maintenance hazard that silently @@ -615,7 +615,7 @@ are unreachable from another TU even if it were linked. Instead the helpers are **DERIVED mechanically at configure time** from the pinned upstream source: -- **`cmake/generate-tts-upstream.cmake`** — reads `${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp`, keeps +- **`llama/cmake/generate-tts-upstream.cmake`** — reads `${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp`, keeps the pre-`main()` span (the DSP `fill_hann_window`/`irfft`/`fold`/`embd_to_audio`, the prompt/text helpers incl. `process_text`'s number-to-words, the `outetts_version` enum), strips `static` from the handful the JNI engine calls (giving them external linkage), and extracts the two hard-coded @@ -1032,7 +1032,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in - The `server` package is a dedicated top layer in the ArchUnit `layeredArchitecture` rule (the only layer allowed to access the root `Api`); `noInternalJdkImports` carries an explicit exception for the supported `com.sun.net.httpserver` (the exported `jdk.httpserver` module, which `module-info.java` `requires`). See README "OpenAI-compatible HTTP server". **Native layer** (`src/main/cpp/`): -- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,650 lines; 33 native methods (29 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`). +- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,650 lines; 34 native methods (30 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`). - `utils.hpp` — Helper utilities (format helpers, argv stripping, token-piece serialisation). - `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable. - `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`. diff --git a/TODO.md b/TODO.md index b45414ba..a6191748 100644 --- a/TODO.md +++ b/TODO.md @@ -91,9 +91,9 @@ workflow in `.github/workflows/`). It contributes to the `mergeable_state: block ### Upstream PR submissions — drop the carried patches (open) -Six of the eight `patches/` are upstream-submittable verbatim; each accepted PR (once the pin is -bumped past it) deletes a patch from the bump checklist. (`0003`/`0004` are carries of already-open -upstream PRs #22393/#23116 — they drop automatically when those merge.) +Six of the seven `patches/` are upstream-submittable verbatim; each accepted PR (once the pin is +bumped past it) deletes a patch from the bump checklist. (`0003` is a carry of an already-open +upstream PR #22393 — it drops automatically when that merges.) - **`0001` Windows arg-parse embed guard** (against #24779): `common_params_parse` trusts the caller's argv; `common_params_parse_main()` keeps the standalone tools' UTF-8 recovery. Ship with the @@ -101,13 +101,13 @@ upstream PRs #22393/#23116 — they drop automatically when those merge.) host process line). - **`0002` preserve caller load-progress callback** (b9789 regression: server clobbers `params_base.load_progress_callback`). -- **`0005` recurrent near-prompt-end checkpoints** (agentic checkpoint starvation on recurrent/hybrid - models; complements upstream #24035/#24899/#24891). - **`0006` embeddable `llama_server`** (no process signal handlers, forwarded-argv parse, out-of-band shutdown). - **`0007` `llama_server_attach`** (HTTP frontend on an existing `server_context`). - **`0008` `LLAMA_SERVER_WORKER_CMD` router worker override** (also useful for containerized/wrapped deployments). +- **`0009` guard `posix_spawn_file_actions_addchdir_np` on old glibc** (b10154 cross-compile break on + manylinux2014 / glibc 2.17; adds a `__GLIBC_PREREQ(2, 29)` probe — submittable to sheredom/subprocess.h). ### llama.cpp upstream feature exposure (queued, deferred by policy) @@ -191,7 +191,7 @@ real arm64 hardware and the Adreno/OpenCL flavor. Treat LLaMAndroid as prior art - **Evaluate GraalVM Native Image as an alternative distribution target.** Reference: [GraalVM Native Image](https://www.graalvm.org/latest/reference-manual/native-image/). The pure-Java sibling projects in the README's "Similar Projects" list (mukel's `llama3.java` / `gemma4.java` / `gptoss.java` / `qwen35.java` / `nemotron3.java`) demonstrate that single-jar, no-JNI Java inference is viable for individual model architectures. Native Image opens an orthogonal direction for THIS project: AOT-compile the Java layer + JNI bridge to a self-contained binary that bundles the libjllama.so (or per-OS equivalent) and starts in milliseconds without a JVM, which would make jllama usable in CLI tools, serverless functions, and short-lived processes where JVM startup is the dominant cost. **What to investigate before committing**: - - **JNI-loading shape.** Native Image supports JNI but requires `--enable-native-access=ALL-UNNAMED` + reflection/JNI configuration files (`reflect-config.json`, `jni-config.json`, `resource-config.json`) describing every class/method/field reachable across the JNI boundary. The 17 native methods in `jllama.cpp` plus the JNI-side `FindClass` / `GetFieldID` / `GetMethodID` calls at `JNI_OnLoad` need to be mapped. The GraalVM tracing agent (`-agentlib:native-image-agent=config-output-dir=...`) can auto-generate the config during a representative test run, but the `LlamaLoader` JAR-extraction path needs at least one resource-config rule for `net/ladenthin/llama/{OS}/{ARCH}/lib*.so`. + - **JNI-loading shape.** Native Image supports JNI but requires `--enable-native-access=ALL-UNNAMED` + reflection/JNI configuration files (`reflect-config.json`, `jni-config.json`, `resource-config.json`) describing every class/method/field reachable across the JNI boundary. The 34 native methods in `jllama.cpp` plus the JNI-side `FindClass` / `GetFieldID` / `GetMethodID` calls at `JNI_OnLoad` need to be mapped. The GraalVM tracing agent (`-agentlib:native-image-agent=config-output-dir=...`) can auto-generate the config during a representative test run, but the `LlamaLoader` JAR-extraction path needs at least one resource-config rule for `net/ladenthin/llama/{OS}/{ARCH}/lib*.so`. - **Native-library packaging.** The current `LlamaLoader` extracts the OS-specific `.so`/`.dll`/`.dylib` from the JAR to a tmp dir at first use. Native Image needs the same file at AOT-execution time, so either (a) ship the native lib alongside the produced binary as a sidecar file and adjust `LlamaLoader` to find it on the same directory, or (b) embed the native lib as a resource and keep the existing extract-to-tmpdir flow (which Native Image supports via `resource-config.json`). - **CUDA / Metal / OpenCL backend selection.** Today the choice between CPU-only / `cuda13-linux-x86-64` / `opencl-android-aarch64` JARs is at Maven-classifier time. Native Image would need either one binary per backend (multiplying the release matrix) or a runtime selector inside `LlamaLoader` that picks among bundled backend libs. The latter is a bigger refactor. - **Startup-time benchmark to justify the work.** Measure cold-start of a current java-llama.cpp `LlamaModel(new ModelParameters().setModel("...").setNPredict(1))` invocation: how much is JVM startup + class load vs JNI load + model parse + tokenize + 1 token? If JVM startup is < 10 % of cold-start, Native Image yields little. If JVM startup is > 50 %, it's a clear win for CLI / serverless use cases. @@ -274,8 +274,8 @@ One-liners for the sections removed from "Open" (full detail: PR #298, CLAUDE.md with GCC 14 (mirroring upstream), runs `ctest` on real ARM (446 tests green), and warms sccache (99.66% hits). Trade-off: glibc floor 2.17 → ~2.39 (same envelope as upstream's ARM binaries); documented in the README classifier table. `build.sh` sccache auto-fetch generalized to aarch64. -- **Generic `patches/` mechanism** — drop `*.patch`/`*.diff` in repo-root `patches/`, applied to the - FetchContent'd llama.cpp source by `cmake/apply-llama-patches.cmake` via the llama.cpp +- **Generic `patches/` mechanism** — drop `*.patch`/`*.diff` in `llama/patches/`, applied to the + FetchContent'd llama.cpp source by `llama/cmake/apply-llama-patches.cmake` via the llama.cpp `PATCH_COMMAND` (cross-platform, idempotent, fail-loud). Covers every C++ build from one place. First patch fixes the Windows JNI arg-parse regression (`1d875b1` → deterministic form `f651b53`). REUSE annotated via `patches/**` glob (`0cffac1`). diff --git a/llama-kotlin/pom.xml b/llama-kotlin/pom.xml index ad644e0a..68d72d99 100644 --- a/llama-kotlin/pom.xml +++ b/llama-kotlin/pom.xml @@ -57,10 +57,9 @@ SPDX-License-Identifier: MIT UTF-8 - + 2.4.10 1.11.0 6.1.2 diff --git a/llama-langchain4j/README.md b/llama-langchain4j/README.md index 9c06d7d2..8628c334 100644 --- a/llama-langchain4j/README.md +++ b/llama-langchain4j/README.md @@ -135,4 +135,4 @@ jina-reranker models the core test jobs already download) — the - **`modelName()`** is ignored since one model is bound per adapter. -Requires Java 17+ (langchain4j 1.x baseline). Targets `langchain4j-core` 1.17.1. +Requires Java 17+ (langchain4j 1.x baseline). Targets `langchain4j-core` 1.18.0.