Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ Current CUDA version: **13.2**

To change the CUDA version, update the following **three** places:

1. **`.github/build_cuda_linux.sh`** — Line 10: `sudo dnf install -y cuda-toolkit-13-2`
2. **`.github/build_cuda_linux.sh`** — Line 12: `-DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc`
1. **`.github/build_cuda_linux.sh`** — Line 16: `sudo dnf install -y cuda-toolkit-13-2`
2. **`.github/build_cuda_linux.sh`** — Line 41: `-DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc`
3. **`llama/pom.xml`** — The `<classifier>` tag in the `cuda` jar execution: `cuda13-linux-x86-64`

Also update the header comment in `build_cuda_linux.sh` and the job name in `.github/workflows/release.yaml` for clarity.
Expand Down Expand Up @@ -557,15 +557,15 @@ are kept jllama-only) is explained in the cross-repo status under "Deliberate no

The fetched llama.cpp source is patched before it compiles, via a generic mechanism:

- **`patches/`** (repo root) — drop any number of `*.patch` / `*.diff` files here. They are applied
- **`llama/patches/`** — drop any number of `*.patch` / `*.diff` files here. They are applied
in **filename order** (use a numeric prefix, e.g. `0001-`, `0002-`), so keep them independent or
ordered. Each must be a `git apply`-compatible unified diff with paths relative to the llama.cpp
source root (`a/common/arg.cpp` / `b/common/arg.cpp`, i.e. `-p1`).
- **`cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on
- **`llama/cmake/apply-llama-patches.cmake`** — the applier. Cross-platform (`cmake -P`, so identical on
Linux/macOS/Windows), **idempotent** (`git apply --reverse --check` skips already-applied patches
so a reconfigure never double-applies) and **fail-loud** (a patch that no longer applies aborts
the configure — a stale patch can't be silently dropped from a release build).
- **`CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it
- **`llama/CMakeLists.txt`** — wired as the llama.cpp `FetchContent_Declare(... PATCH_COMMAND ...)`, so it
runs for **every** C++ build (all CI jobs *and* local `cmake -B build`) from one place — no
per-build-step plumbing.

Expand Down Expand Up @@ -604,7 +604,7 @@ prefill behavior shows up, re-check `tools/server/server-context.cpp`'s `create_
(`test_reasoning_budget_tokens_per_request` / `test_reasoning_budget_message_per_request`,
byte-identical body). No local patch needed — the tree already matches what `0004` used to add.

## OuteTTS build-time extraction (`cmake/generate-tts-upstream.cmake`)
## OuteTTS build-time extraction (`llama/cmake/generate-tts-upstream.cmake`)

The `TextToSpeech` native pipeline reuses llama.cpp's OuteTTS helpers (`tools/tts/tts.cpp`)
**without hand-copying them**. A verbatim copy would be a DRY/maintenance hazard that silently
Expand All @@ -615,7 +615,7 @@ are unreachable from another TU even if it were linked.

Instead the helpers are **DERIVED mechanically at configure time** from the pinned upstream source:

- **`cmake/generate-tts-upstream.cmake`** — reads `${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp`, keeps
- **`llama/cmake/generate-tts-upstream.cmake`** — reads `${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp`, keeps
the pre-`main()` span (the DSP `fill_hann_window`/`irfft`/`fold`/`embd_to_audio`, the prompt/text
helpers incl. `process_text`'s number-to-words, the `outetts_version` enum), strips `static` from
the handful the JNI engine calls (giving them external linkage), and extracts the two hard-coded
Expand Down Expand Up @@ -1032,7 +1032,7 @@ If the local check passes (`BUILD SUCCESS`), the `mvn package` job in
- The `server` package is a dedicated top layer in the ArchUnit `layeredArchitecture` rule (the only layer allowed to access the root `Api`); `noInternalJdkImports` carries an explicit exception for the supported `com.sun.net.httpserver` (the exported `jdk.httpserver` module, which `module-info.java` `requires`). See README "OpenAI-compatible HTTP server".

**Native layer** (`src/main/cpp/`):
- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,650 lines; 33 native methods (29 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`).
- `jllama.cpp` — JNI implementation bridging Java calls to llama.cpp. ~1,650 lines; 34 native methods (30 `LlamaModel` + 3 `TextToSpeech` + 1 `LlamaQuantizer`).
- `utils.hpp` — Helper utilities (format helpers, argv stripping, token-piece serialisation).
- `json_helpers.hpp` — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable.
- `jni_helpers.hpp` — JNI bridge helpers (handle management + server orchestration). Includes `json_helpers.hpp`.
Expand Down
16 changes: 8 additions & 8 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,23 +91,23 @@ workflow in `.github/workflows/`). It contributes to the `mergeable_state: block

### Upstream PR submissions — drop the carried patches (open)

Six of the eight `patches/` are upstream-submittable verbatim; each accepted PR (once the pin is
bumped past it) deletes a patch from the bump checklist. (`0003`/`0004` are carries of already-open
upstream PRs #22393/#23116they drop automatically when those merge.)
Six of the seven `patches/` are upstream-submittable verbatim; each accepted PR (once the pin is
bumped past it) deletes a patch from the bump checklist. (`0003` is a carry of an already-open
upstream PR #22393 — it drops automatically when that merges.)

- **`0001` Windows arg-parse embed guard** (against #24779): `common_params_parse` trusts the caller's
argv; `common_params_parse_main()` keeps the standalone tools' UTF-8 recovery. Ship with the
standalone-safe repro (synthetic argv discarded on Windows because `GetCommandLineW()` returns the
host process line).
- **`0002` preserve caller load-progress callback** (b9789 regression: server clobbers
`params_base.load_progress_callback`).
- **`0005` recurrent near-prompt-end checkpoints** (agentic checkpoint starvation on recurrent/hybrid
models; complements upstream #24035/#24899/#24891).
- **`0006` embeddable `llama_server`** (no process signal handlers, forwarded-argv parse, out-of-band
shutdown).
- **`0007` `llama_server_attach`** (HTTP frontend on an existing `server_context`).
- **`0008` `LLAMA_SERVER_WORKER_CMD` router worker override** (also useful for containerized/wrapped
deployments).
- **`0009` guard `posix_spawn_file_actions_addchdir_np` on old glibc** (b10154 cross-compile break on
manylinux2014 / glibc 2.17; adds a `__GLIBC_PREREQ(2, 29)` probe — submittable to sheredom/subprocess.h).

### llama.cpp upstream feature exposure (queued, deferred by policy)

Expand Down Expand Up @@ -191,7 +191,7 @@ real arm64 hardware and the Adreno/OpenCL flavor. Treat LLaMAndroid as prior art
- **Evaluate GraalVM Native Image as an alternative distribution target.** Reference: [GraalVM Native Image](https://www.graalvm.org/latest/reference-manual/native-image/). The pure-Java sibling projects in the README's "Similar Projects" list (mukel's `llama3.java` / `gemma4.java` / `gptoss.java` / `qwen35.java` / `nemotron3.java`) demonstrate that single-jar, no-JNI Java inference is viable for individual model architectures. Native Image opens an orthogonal direction for THIS project: AOT-compile the Java layer + JNI bridge to a self-contained binary that bundles the libjllama.so (or per-OS equivalent) and starts in milliseconds without a JVM, which would make jllama usable in CLI tools, serverless functions, and short-lived processes where JVM startup is the dominant cost.

**What to investigate before committing**:
- **JNI-loading shape.** Native Image supports JNI but requires `--enable-native-access=ALL-UNNAMED` + reflection/JNI configuration files (`reflect-config.json`, `jni-config.json`, `resource-config.json`) describing every class/method/field reachable across the JNI boundary. The 17 native methods in `jllama.cpp` plus the JNI-side `FindClass` / `GetFieldID` / `GetMethodID` calls at `JNI_OnLoad` need to be mapped. The GraalVM tracing agent (`-agentlib:native-image-agent=config-output-dir=...`) can auto-generate the config during a representative test run, but the `LlamaLoader` JAR-extraction path needs at least one resource-config rule for `net/ladenthin/llama/{OS}/{ARCH}/lib*.so`.
- **JNI-loading shape.** Native Image supports JNI but requires `--enable-native-access=ALL-UNNAMED` + reflection/JNI configuration files (`reflect-config.json`, `jni-config.json`, `resource-config.json`) describing every class/method/field reachable across the JNI boundary. The 34 native methods in `jllama.cpp` plus the JNI-side `FindClass` / `GetFieldID` / `GetMethodID` calls at `JNI_OnLoad` need to be mapped. The GraalVM tracing agent (`-agentlib:native-image-agent=config-output-dir=...`) can auto-generate the config during a representative test run, but the `LlamaLoader` JAR-extraction path needs at least one resource-config rule for `net/ladenthin/llama/{OS}/{ARCH}/lib*.so`.
- **Native-library packaging.** The current `LlamaLoader` extracts the OS-specific `.so`/`.dll`/`.dylib` from the JAR to a tmp dir at first use. Native Image needs the same file at AOT-execution time, so either (a) ship the native lib alongside the produced binary as a sidecar file and adjust `LlamaLoader` to find it on the same directory, or (b) embed the native lib as a resource and keep the existing extract-to-tmpdir flow (which Native Image supports via `resource-config.json`).
- **CUDA / Metal / OpenCL backend selection.** Today the choice between CPU-only / `cuda13-linux-x86-64` / `opencl-android-aarch64` JARs is at Maven-classifier time. Native Image would need either one binary per backend (multiplying the release matrix) or a runtime selector inside `LlamaLoader` that picks among bundled backend libs. The latter is a bigger refactor.
- **Startup-time benchmark to justify the work.** Measure cold-start of a current java-llama.cpp `LlamaModel(new ModelParameters().setModel("...").setNPredict(1))` invocation: how much is JVM startup + class load vs JNI load + model parse + tokenize + 1 token? If JVM startup is < 10 % of cold-start, Native Image yields little. If JVM startup is > 50 %, it's a clear win for CLI / serverless use cases.
Expand Down Expand Up @@ -274,8 +274,8 @@ One-liners for the sections removed from "Open" (full detail: PR #298, CLAUDE.md
with GCC 14 (mirroring upstream), runs `ctest` on real ARM (446 tests green), and warms sccache
(99.66% hits). Trade-off: glibc floor 2.17 → ~2.39 (same envelope as upstream's ARM binaries);
documented in the README classifier table. `build.sh` sccache auto-fetch generalized to aarch64.
- **Generic `patches/` mechanism** — drop `*.patch`/`*.diff` in repo-root `patches/`, applied to the
FetchContent'd llama.cpp source by `cmake/apply-llama-patches.cmake` via the llama.cpp
- **Generic `patches/` mechanism** — drop `*.patch`/`*.diff` in `llama/patches/`, applied to the
FetchContent'd llama.cpp source by `llama/cmake/apply-llama-patches.cmake` via the llama.cpp
`PATCH_COMMAND` (cross-platform, idempotent, fail-loud). Covers every C++ build from one place.
First patch fixes the Windows JNI arg-parse regression (`1d875b1` → deterministic form `f651b53`).
REUSE annotated via `patches/**` glob (`0cffac1`).
Expand Down
7 changes: 3 additions & 4 deletions llama-kotlin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,9 @@ SPDX-License-Identifier: MIT

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!-- One minor behind the latest stable Kotlin on purpose: Kotlin consumers can
read metadata at most one minor version ahead of their own compiler, so
compiling with 2.2.x keeps this artifact consumable from Kotlin 2.1+
projects (typical Android Studio installs) instead of forcing 2.2+. -->
<!-- Kotlin consumers can read metadata at most one minor version ahead of
their own compiler, so compiling with Kotlin 2.4 keeps this artifact
consumable from Kotlin 2.3+ projects (typical Android Studio installs). -->
<kotlin.version>2.4.10</kotlin.version>
<kotlinx.coroutines.version>1.11.0</kotlinx.coroutines.version>
<junit.version>6.1.2</junit.version>
Expand Down
2 changes: 1 addition & 1 deletion llama-langchain4j/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,4 +135,4 @@ jina-reranker models the core test jobs already download) — the

- **`modelName()`** is ignored since one model is bound per adapter.

Requires Java 17+ (langchain4j 1.x baseline). Targets `langchain4j-core` 1.17.1.
Requires Java 17+ (langchain4j 1.x baseline). Targets `langchain4j-core` 1.18.0.
Loading