diff --git a/README.md b/README.md
index 575754ae..879ef761 100644
--- a/README.md
+++ b/README.md
@@ -1,626 +1,321 @@
-# GPULlama3.java powered by TornadoVM [](https://github.com/beehive-lab/GPULlama3.java/actions/workflows/build-and-run.yml) [](https://central.sonatype.com/artifact/io.github.beehive-lab/gpu-llama3)
+# GPULlama3.java โ LLM inference & serving for the JVM, on any GPU
-
-
-
+[](https://github.com/beehive-lab/GPULlama3.java/actions/workflows/build-and-run.yml)
+[](https://central.sonatype.com/artifact/io.github.beehive-lab/gpu-llama3)
+
+
[](https://docs.langchain4j.dev/)
-
-
-[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl)
-[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx)
-[](https://deepwiki.com/beehive-lab/GPULlama3.java)
+
+
+
+[](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl)
+[](https://deepwiki.com/beehive-lab/GPULlama3.java)
+[](LICENSE)
-----------
+
|
-
-Llama3 models written in native Java automatically accelerated on GPUs with TornadoVM.
-Runs Llama3 inference efficiently using TornadoVM's GPU acceleration.
-
-Currently, supports Llama3, Mistral, Devstral 2, Qwen2.5, Qwen3, Phi-3, IBM Granite 3.2+ and IBM Granite 4.0 models in the GGUF format.
-Also, it is used as GPU inference engine in
-Quarkus
-and
-LangChain4J.
-
-Builds on Llama3.java by Alfonsoยฒ Peterssen.
-Previous integration of TornadoVM and Llama2 it can be found in llama2.tornadovm.
- |
-
-
+
------------
-##  Integration with LangChain4j
+### Think vLLM โ but pure Java, and it runs on **any** GPU.
-Since **LangChain4j v1.7.1**, `GPULlama3.java` is officially supported as a **model provider**.
-This means you can directly use *GPULlama3.java* inside your LangChain4j applications without extra glue code, just run on your GPU.
+**GPULlama3.java** is a JVM-native LLM inference and serving engine. You write and ship plain Java; [**TornadoVM**](https://github.com/beehive-lab/TornadoVM) JIT-compiles the hot transformer kernels to **CUDA, PTX, OpenCL, or Apple Metal** at runtime โ no JNI glue, no second toolchain, no native rebuild per GPU.
-๐ Learn more: [LangChain4j Documentation](https://docs.langchain4j.dev/)
+One `.jar` runs the same model on **NVIDIA, Intel, AMD, and Apple Silicon**, from a laptop to an RTX 5090.
-[Example agentic workflows with GPULlama3.java + LangChain4j ๐](https://github.com/mikepapadim/devoxx25-demo-gpullama3-langchain4j/tree/main)
+Serve it behind an **OpenAI-compatible API**, embed it in **LangChain4j** or **Quarkus**, or run it from the CLI in one line.
-How to use:
-```java
-GPULlama3ChatModel model = GPULlama3ChatModel.builder()
- .modelPath(modelPath)
- .temperature(0.9) // more creative
- .topP(0.9) // more variety
- .maxTokens(2048)
- .onGPU(Boolean.TRUE) // if false, runs on CPU though a lightweight implementation of llama3.java
- .build();
-```
------------
-#### **[Interactive-mode]** Running on a RTX 5090 with nvtop on bottom to track GPU utilization and memory usage.
+ |
+
+
-
+Builds on [Llama3.java](https://github.com/mukel/llama3.java) by [Alfonsoยฒ Peterssen](https://github.com/mukel). Earlier Llama2 work: [llama2.tornadovm](https://github.com/mikepapadim/llama2.tornadovm.java).
-----------
-## Setup & Configuration
+## Why GPULlama3.java
-### Prerequisites
+- ๐ฆ **Pure Java, all the way down.** Transformer kernels are written in Java and accelerated by TornadoVM โ no CUDA C, no hand-written JNI. Debug and build with the toolchain you already have.
+- ๐ **Write once, run on any GPU.** NVIDIA (CUDA / PTX), Intel & AMD (OpenCL), Apple Silicon (Metal). Backend is auto-detected from your TornadoVM SDK โ switch with a flag, not a rebuild.
+- ๐ **Drop-in for the Java AI stack.** Official [LangChain4j](https://docs.langchain4j.dev/integrations/language-models/gpullama3-java) provider (since v1.7.1) and [Quarkus](https://docs.quarkiverse.io/quarkus-langchain4j/dev/gpullama3-chat-model.html) inference engine.
+- โก **Built to serve.** OpenAI-compatible HTTP server, llama-bench-style benchmarking, and tensor-core (MMA) batch prefill (see [Serving](#-serving-openai-compatible-preview)).
+- ๐ฆ **Many models, one runtime.** Llama 3, Mistral, Qwen 2.5 / Qwen 3, Phi-3, IBM Granite 3.3 / 4.0, DeepSeek-R1-Distill โ all in GGUF.
-Ensure you have the following installed and configured:
-
-- **Java 21**: Required for Vector API support & TornadoVM.
-- [TornadoVM](https://github.com/beehive-lab/TornadoVM) with an OpenCL, PTX, CUDA, or Metal backend. `llama-tornado`/`llamaTornado` auto-detect whichever backend your installed SDK was built with.
- - The CUDA backend requires a TornadoVM build that includes the CUDA backend from [TornadoVM PR #861](https://github.com/beehive-lab/TornadoVM/pull/861). This project currently builds against TornadoVM `5.0.0-jdk21-dev`.
-- GCC/G++ 13 or newer: Required to build and run TornadoVM native components.
-
-### Install, Build, and Run
-
-```bash
-# Clone the repository with all submodules
-git clone https://github.com/beehive-lab/GPULlama3.java.git
-```
-
-#### Install the TornadoVM SDK on Linux or macOS
-
-Ensure that your JAVA_HOME points to a supported JDK before using the SDK. Download an SDK package matching your OS, architecture, and accelerator backend (opencl, ptx).
-TornadoVM is distributed through our [**official website**](https://www.tornadovm.org/downloads) and **SDKMAN!**. Install a version that matches your OS, architecture, and accelerator backend.
-
-All TornadoVM SDKs are available on the [SDKMAN! TornadoVM page](https://sdkman.io/sdks/tornadovm/).
+-----------
-#### SDKMAN! Installation (Recommended)
+## โฑ๏ธ Quickstart (60 seconds)
-##### Install SDKMAN! if not installed already
-```bash
-curl -s "https://get.sdkman.io" | bash
-source "$HOME/.sdkman/bin/sdkman-init.sh"
-sdk version
-```
-##### Install TornadoVM via SDKMAN!
```bash
+# 1. Install a TornadoVM SDK (bundles the GPU runtime)
+curl -s "https://get.sdkman.io" | bash && source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install tornadovm
-```
+tornado --devices # confirm your GPU is listed
-#### Verify TornadoVM is Installed Correctly
-```bash
-tornado --devices
+# 2. Run a model on the GPU โ no build required, via JBang
+jbang gpullama3@beehive-lab -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain GPU acceleration in one sentence."
```
-----------
-
-### TornadoVM-Accelerated Inference Performance and Optimization Status
-We are at the early stages of Java entering the AI world with features added to the JVM that enable faster execution such as GPU acceleration, Vector acceleration, high-performance access to off-heap memory and others.
+Grab a ready-to-run model from the [Hugging Face collections](#-model-collections) below.
+-----------
-| Vendor / Backend | Hardware | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | Optimizations |
-|:----------------------------:|:------------:|:---------------------:|:---------------------:|:-------------:|
-| | | **FP16** | **FP16** | **Support** |
-| **NVIDIA / OpenCL-PTX** | RTX 3070 | 66 tokens/s | 55.46 tokens/s | โ
|
-| | RTX 4090 | 86.11 tokens/s | 75.32 tokens/s | โ
|
-| | RTX 5090 | 117.65 tokens/s | 112.68 tokens/s | โ
|
-| | L4 Tensor | 52.96 tokens/s | 22.68 tokens/s | โ
|
-| **Intel / OpenCL** | Arc A770 | 15.65 tokens/s | 7.02 tokens/s | (WIP) |
-| **Apple Silicon / OpenCL** | M3 Pro | 14.04 tokens/s | 6.78 tokens/s | (WIP) |
-| | M4 Pro | 16.77 tokens/s | 8.56 tokens/s | (WIP) |
-| **AMD / OpenCL** | Radeon RX | (WIP) | (WIP) | (WIP) |
+## ๐งฉ Serving: OpenAI-compatible (preview)
-##### Apple Silicon Support
+GPULlama3.java is growing into a **serving engine** โ the vLLM-style path for the JVM:
-TornadoVM 4.0 includes a native [Metal](https://developer.apple.com/metal/) backend, enabling GPU-accelerated inference on Apple Silicon.
+- ๐ **OpenAI-compatible server** โ `llama-tornado --server` exposes `/v1/chat/completions` and `/v1/completions` with streaming and zero external dependencies. Point any OpenAI client at `localhost`. See [OpenAI-compatible server](#-openai-compatible-server---server) below.
+- ๐ฏ **Tensor-core (MMA) batch prefill** on the CUDA backend, FP16 & Q8_0 โ `--with-prefill-decode --batch-prefill-size N`.
+- ๐ **llama-bench-style benchmarking** โ `llama-tornado --bench` reports a pp/tg matrix with avgยฑstddev in md/csv/json/jsonl/sql. See [Benchmarking](#-benchmarking---bench-llama-bench-style) below.
+- ๐งฎ **On-device greedy sampling** *(landing next)* โ argmax on the GPU keeps logits device-side, cutting deviceโhost traffic by ~500ร per token. ([PR #134](https://github.com/beehive-lab/GPULlama3.java/pull/134))
+- ๐ **Static batched decode** *(landing next)* โ B independent sequences per step for up to **41ร aggregate throughput** (Llama & Qwen3). ([PR #129](https://github.com/beehive-lab/GPULlama3.java/pull/129))
-----------
-## ๐ฆ Maven Dependency
-You can add **GPULlama3.java** directly to your Maven project by including the following dependency in your `pom.xml`:
+## LangChain4j & Quarkus
-**JDK 21:**
-```xml
-
- io.github.beehive-lab
- gpu-llama3
- 0.5.0
-
+Since **LangChain4j v1.7.1**, `GPULlama3.java` is an officially supported **model provider** โ no glue code, GPU-accelerated out of the box.
+
+```java
+GPULlama3ChatModel model = GPULlama3ChatModel.builder()
+ .modelPath(modelPath)
+ .temperature(0.9) // more creative
+ .topP(0.9) // more variety
+ .maxTokens(2048)
+ .onGPU(Boolean.TRUE) // false โ lightweight CPU llama3.java
+ .build();
```
-**JDK 25:**
+๐ [LangChain4j docs](https://docs.langchain4j.dev/) ยท ๐ [Agentic workflow demo](https://github.com/mikepapadim/devoxx25-demo-gpullama3-langchain4j/tree/main)
+
+### ๐ฆ Maven
+
```xml
+
io.github.beehive-lab
gpu-llama3
- 0.5.0-jdk25
+ 0.5.0
-```
-
-## โ Integration with Your Java Codebase or Tools
-
-To integrate it into your codebase or IDE (e.g., IntelliJ) or custom build system (like IntelliJ, Maven, or Gradle), use the `--show-command` flag.
-This flag shows the exact Java command with all JVM flags that are being invoked under the hood to enable seamless execution on GPUs with TornadoVM.
-Hence, it makes it simple to replicate or embed the invoked flags in any external tool or codebase.
-```bash
-llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "tell me a joke" --show-command
+
```
-
-๐ Click to see the JVM configuration
-
-```java
-/home/mikepapadim/.sdkman/candidates/java/current/bin/java \
- -server \
- -XX:+UnlockExperimentalVMOptions \
- -XX:+EnableJVMCI \
- -Xms20g -Xmx20g \
- --enable-preview \
- -Djava.library.path=/home/mikepapadim/manchester/TornadoVM/bin/sdk/lib \
- -Djdk.module.showModuleResolution=false \
- --module-path .:/home/mikepapadim/manchester/TornadoVM/bin/sdk/share/java/tornado \
- -Dtornado.load.api.implementation=uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph \
- -Dtornado.load.runtime.implementation=uk.ac.manchester.tornado.runtime.TornadoCoreRuntime \
- -Dtornado.load.tornado.implementation=uk.ac.manchester.tornado.runtime.common.Tornado \
- -Dtornado.load.annotation.implementation=uk.ac.manchester.tornado.annotation.ASMClassVisitor \
- -Dtornado.load.annotation.parallel=uk.ac.manchester.tornado.api.annotations.Parallel \
- -Dtornado.tvm.maxbytecodesize=65536 \
- -Duse.tornadovm=true \
- -Dtornado.threadInfo=false \
- -Dtornado.debug=false \
- -Dtornado.fullDebug=false \
- -Dtornado.printKernel=false \
- -Dtornado.print.bytecodes=false \
- -Dtornado.device.memory=7GB \
- -Dtornado.profiler=false \
- -Dtornado.log.profiler=false \
- -Dtornado.profiler.dump.dir=/home/mikepapadim/repos/gpu-llama3.java/prof.json \
- -Dtornado.enable.fastMathOptimizations=true \
- -Dtornado.enable.mathOptimizations=false \
- -Dtornado.enable.nativeFunctions=fast \
- -Dtornado.loop.interchange=true \
- -Dtornado.eventpool.maxwaitevents=32000 \
- "-Dtornado.opencl.compiler.flags=-cl-denorms-are-zero -cl-no-signed-zeros -cl-finite-math-only" \
- --upgrade-module-path /home/mikepapadim/manchester/TornadoVM/bin/sdk/share/java/graalJars \
- @/home/mikepapadim/manchester/TornadoVM/bin/sdk/etc/exportLists/common-exports \
- @/home/mikepapadim/manchester/TornadoVM/bin/sdk/etc/exportLists/opencl-exports \
- --add-modules ALL-SYSTEM,tornado.runtime,tornado.annotation,tornado.drivers.common,tornado.drivers.opencl \
- -cp /home/mikepapadim/repos/gpu-llama3.java/target/gpu-llama3-1.0-SNAPSHOT.jar \
- org.beehive.gpullama3.LlamaApp \
- -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- --temperature 0.1 \
- --top-p 0.95 \
- --seed 1746903566 \
- --max-tokens 512 \
- --stream true \
- --echo false \
- -p "tell me a joke" \
- --instruct
-```
-
-
-
-----------
-The above model can we swapped with one of the other models, such as `beehive-llama-3.2-3b-instruct-fp16.gguf` or `beehive-llama-3.2-8b-instruct-fp16.gguf`, depending on your needs.
-Check models below.
+#### **[Interactive mode]** โ RTX 5090, with `nvtop` tracking GPU utilization and memory
------------
+
-## ๐ Running with JBang (Pure Java CLI)
+-----------
-You can run llama-tornado as a pure Java script using [JBang](https://www.jbang.dev/) without building or installing anything. This provides a simple, script-like experience similar to [Jlama's CLI](https://github.com/tjake/Jlama).
+## ๐ ๏ธ Install & build
-### Prerequisites for JBang
+### Prerequisites
-1. **Install JBang**: Follow the [JBang installation guide](https://www.jbang.dev/download/)
-2. **TornadoVM SDK**: You still need TornadoVM installed and `TORNADOVM_HOME` environment variable set (see Setup section above)
+- **Java 21** โ required for the Vector API & TornadoVM (Java 25 supported via the `-jdk25` artifact / `llamaTornado` script).
+- **[TornadoVM](https://github.com/beehive-lab/TornadoVM)** with an OpenCL, PTX, CUDA, or Metal backend. `llama-tornado`/`llamaTornado` auto-detect whichever backend your installed SDK was built with.
+- **GCC/G++ 13+** โ to build TornadoVM's native components.
-### Quick Start with JBang
+### Get TornadoVM (SDKMAN!, recommended)
-Use from catalog:
+TornadoVM is distributed via the [official website](https://www.tornadovm.org/downloads) and [SDKMAN!](https://sdkman.io/sdks/tornadovm/). Pick a package matching your OS, architecture, and backend (opencl, ptx).
```bash
-# Install JBang (if not already installed)
-curl -Ls https://sh.jbang.dev | bash -s - app setup
-
-# Run GPULlama3.java CLI
-jbang gpullama3@beehive-lab -m model.gguf -p "Tell me a joke"
-
-# Or install it as a command
-jbang app install gpullama3@beehive-lab
-gpullama3 -m model.gguf -p "Hello!"
+sdk install tornadovm
+tornado --devices # verify
```
-or the local:
-```bash
-# Basic usage - interactive chat mode
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf --interactive
-
-# Single instruction mode
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain quantum computing"
-# With TornadoVM GPU acceleration
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- -p "Tell me a joke" --use-tornadovm true
+### Clone this repo
-# Custom generation parameters
-jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf \
- -p "Write a short story" \
- --temperature 0.7 \
- --top-p 0.9 \
- --max-tokens 512
+```bash
+git clone https://github.com/beehive-lab/GPULlama3.java.git
```
-----------
-## Collection of Tested Models
-
-### Llama3.2 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/llama3-gpullama3java)
-
-### IBM Granite 4.0 Collection
-[https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java](https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java)
-
-
-### IBM Granite 3.3 Collection
-[https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java](https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java)
-
-### Qwen 2.5 Collection
-[https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java)
-
-### Qwen 3 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-3-gpullama3java)
-
-### Phi-3 Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/phi-3-gpullama3java)
-
-### Mistral Collection
-[https://huggingface.co/collections/beehive-lab/llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/mistral-gpullama3java)
-
-### DeepSeek-R1-Distill-Qwen Collection
-[https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java](https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java)
-
------------
-
-## Running `llama-tornado`
+## โถ๏ธ Running the CLI
-To execute Llama3, or Mistral models with TornadoVM on GPUs use the `llama-tornado` script with the `--gpu` flag.
-
-The TornadoVM backend (OpenCL, PTX, CUDA, or Metal) is auto-detected from your installed
-TornadoVM SDK (`TORNADOVM_HOME/etc/tornado.backend`) - no need to select it manually.
-
-### Usage Examples
-
-#### Basic Inference
-Run a model with a text prompt:
+Use the `llama-tornado` script with `--gpu`. The backend (OpenCL, PTX, CUDA, or Metal) is auto-detected from
+your installed TornadoVM SDK (`TORNADOVM_HOME/etc/tornado.backend`) โ no need to select it manually. If your
+SDK was built with more than one backend, force one with `--opencl`, `--ptx`, `--cuda` (NVIDIA), or `--metal`
+(Apple Silicon); forcing a backend that isn't part of the installed SDK errors out.
```bash
-./llama-tornado --gpu --verbose-init --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "Explain the benefits of GPU acceleration."
+# Basic GPU inference โ backend auto-detected
+./llama-tornado --gpu --verbose-init \
+ --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "Explain the benefits of GPU acceleration."
+
+# Force a specific backend (only needed for multi-backend SDKs)
+./llama-tornado --gpu --cuda \
+ --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "Explain the benefits of GPU acceleration."
```
-The script prints which backend it detected, e.g. `Detected TornadoVM backend: cuda (from .../etc/tornado.backend)`.
-To run against a different backend, point `TORNADOVM_HOME` at an SDK built for that backend instead - or,
-if your SDK was built with more than one backend (e.g. `cuda-opencl`), pass `--opencl`/`--ptx`/`--cuda`/`--metal`
-to force one of the installed backends without needing a separate SDK:
+Swap in any tested model โ e.g. `beehive-llama-3.2-3b-instruct-fp16.gguf` or `...-8b-...`.
-```bash
-./llama-tornado --gpu --opencl --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "Explain the benefits of GPU acceleration."
-```
+### `llamaTornado` โ zero-dependency Java 25 script
-These flags error out if the requested backend isn't part of the installed SDK.
+Same backend auto-detection as `llama-tornado`. A single-file Java 25 launcher that replaces the Python
+script (needs `java 25+` on your PATH):
-#### GPU Execution (FP16 Model)
-Enable GPU acceleration with Q8_0 quantization:
```bash
-./llama-tornado --gpu --verbose-init --model beehive-llama-3.2-1b-instruct-fp16.gguf --prompt "tell me a joke"
+./llamaTornado --gpu --verbose-init --metal \
+ --model Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"
```
-#### Running with `llamaTornado` (Java 25 single-file script)
+### ๐ JBang โ run without building
-`llamaTornado` is a zero-dependency Java 25 single-file script that replaces the Python launcher. It requires `java 25+` on your PATH:
+Script-like startup ร la [Jlama](https://github.com/tjake/Jlama), powered by [JBang](https://www.jbang.dev/):
```bash
-./llamaTornado --gpu --verbose-init --model /Users/abien/work/workspaces/llms/Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"
-```
+curl -Ls https://sh.jbang.dev | bash -s - app setup
-Same backend auto-detection as `llama-tornado`, including the `--opencl`/`--ptx`/`--cuda`/`--metal`
-override for SDKs built with more than one backend:
+# From the catalog
+jbang gpullama3@beehive-lab -m model.gguf -p "Tell me a joke"
+jbang app install gpullama3@beehive-lab && gpullama3 -m model.gguf -p "Hello!"
-```bash
-./llamaTornado --gpu --opencl --model /Users/abien/work/workspaces/llms/Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"
+# Or the local script, interactive
+jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf --interactive
```
------------
-
-## ๐ณ Docker
-
-You can run `GPULlama3.java` fully containerized with GPU acceleration enabled via **OpenCL** or **PTX** using pre-built Docker images.
-More information as well as examples to run with the containers are available at [docker-gpullama3.java](https://github.com/beehive-lab/docker-gpullama3.java).
+### ๐ณ Docker
-### ๐ฆ Available Docker Images
+Fully containerized GPU inference via pre-built images ([docker-gpullama3.java](https://github.com/beehive-lab/docker-gpullama3.java)):
-| Backend | Docker Image | Pull Command |
-|--------|---------------|---------------|
-| **OpenCL** | [`beehivelab/gpullama3.java-nvidia-openjdk-opencl`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl) | `docker pull beehivelab/gpullama3.java-nvidia-openjdk-opencl` |
-| **PTX (CUDA)** | [`beehivelab/gpullama3.java-nvidia-openjdk-ptx`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx) | `docker pull beehivelab/gpullama3.java-nvidia-openjdk-ptx` |
-
-#### Example (OpenCL)
+| Backend | Image |
+|---------|-------|
+| **OpenCL** | [`beehivelab/gpullama3.java-nvidia-openjdk-opencl`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-opencl) |
+| **PTX (CUDA)** | [`beehivelab/gpullama3.java-nvidia-openjdk-ptx`](https://hub.docker.com/r/beehivelab/gpullama3.java-nvidia-openjdk-ptx) |
```bash
-docker run --rm -it --gpus all \
- -v "$PWD":/data \
+docker run --rm -it --gpus all -v "$PWD":/data \
beehivelab/gpullama3.java-nvidia-openjdk-opencl \
/gpullama3/GPULlama3.java/llama-tornado \
--gpu --verbose-init \
- --model /data/Llama-3.2-1B-Instruct.FP16.gguf \
- --prompt "Tell me a joke"
-```
------------
-
-## Troubleshooting GPU Memory Issues
-
-### Out of Memory Error
-
-You may encounter an out-of-memory error like:
+ --model /data/Llama-3.2-1B-Instruct.FP16.gguf --prompt "Tell me a joke"
```
-Exception in thread "main" uk.ac.manchester.tornado.api.exceptions.TornadoOutOfMemoryException: Unable to allocate 100663320 bytes of memory.
-To increase the maximum device memory, use -Dtornado.device.memory=GB
-```
-
-This indicates that the default GPU memory allocation (7GB) is insufficient for your model.
-
-### Solution
-
-First, check your GPU specifications. If your GPU has high memory capacity, you can increase the GPU memory allocation using the `--gpu-memory` flag:
-
-```bash
-# For 3B models, try increasing to 15GB
-./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-fp16.gguf --prompt "Tell me a joke" --gpu-memory 15GB
-# For 8B models, you may need even more (20GB or higher)
-./llama-tornado --gpu --model beehive-llama-3.2-8b-instruct-fp16.gguf --prompt "Tell me a joke" --gpu-memory 20GB
-```
+-----------
-### GPU Memory Requirements by Model Size
+## ๐ค Model collections
-| Model Size | Recommended GPU Memory |
-|-------------|------------------------|
-| 1B models | 7GB (default) |
-| 3-7B models | 15GB |
-| 8B models | 20GB+ |
+GGUF models, ready to download:
-**Note**: If you still encounter memory issues, try:
+| Family | Collection |
+|--------|-----------|
+| Llama 3.2 | [llama3-gpullama3java](https://huggingface.co/collections/beehive-lab/llama3-gpullama3java) |
+| IBM Granite 4.0 | [granite-40-language-models](https://huggingface.co/collections/beehive-lab/granite-40-language-models-gpullama3java) |
+| IBM Granite 3.3 | [granite-33-language-models](https://huggingface.co/collections/beehive-lab/granite-33-language-models-gpullama3java) |
+| Qwen 2.5 | [qwen-25-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-25-gpullama3java) |
+| Qwen 3 | [qwen-3-gpullama3java](https://huggingface.co/collections/beehive-lab/qwen-3-gpullama3java) |
+| Phi-3 | [phi-3-gpullama3java](https://huggingface.co/collections/beehive-lab/phi-3-gpullama3java) |
+| Mistral | [mistral-gpullama3java](https://huggingface.co/collections/beehive-lab/mistral-gpullama3java) |
+| DeepSeek-R1-Distill-Qwen | [deepseek-r1-distill-qwen](https://huggingface.co/collections/beehive-lab/deepseek-r1-distill-qwen-gpullama3java) |
-1. Using Q4_0 instead of Q8_0 quantization (requires less memory).
-2. Closing other GPU-intensive applications in your system.
+Formats: GGUF ยท FP16 (full), Q8_0 & Q4_0 (partial).
-----------
-## Command Line Options
-
-Supported command-line options include:
-
-```bash
-cmd โ llama-tornado --help
-usage: llama-tornado [-h] --model MODEL_PATH [--prompt PROMPT]
- [-sp SYSTEM_PROMPT] [--temperature TEMPERATURE]
- [--top-p TOP_P] [--seed SEED] [-n MAX_TOKENS]
- [--stream STREAM] [--echo ECHO] [--suffix SUFFIX] [-i]
- [--instruct] [--server] [--port PORT] [--bench]
- [--bench-args BENCH_ARGS] [--gpu] [--opencl] [--ptx]
- [--cuda] [--metal] [--gpu-memory GPU_MEMORY]
- [--heap-min HEAP_MIN] [--heap-max HEAP_MAX] [--debug]
- [--profiler] [--profiler-dump-dir PROFILER_DUMP_DIR]
- [--print-bytecodes] [--print-threads] [--print-kernel]
- [--full-dump] [--verbose-init] [--show-command]
- [--execute-after-show] [--with-prefill-decode]
- [--batch-prefill-size N] [--cuda-graphs]
- [--opencl-flags OPENCL_FLAGS]
- [--max-wait-events MAX_WAIT_EVENTS] [--verbose]
-
-GPU-accelerated LLM runner using TornadoVM
-(the TornadoVM backend is auto-detected from TORNADOVM_HOME/etc/tornado.backend)
-
-options:
- -h, --help show this help message and exit
- --model MODEL_PATH Path to the LLM gguf file (e.g.,
- Llama-3.2-1B-Instruct-Q8_0.gguf)
-
-LLaMA Configuration:
- --prompt PROMPT Input prompt for the model (default: None)
- -sp, --system-prompt SYSTEM_PROMPT
- System prompt for the model (default: None)
- --temperature TEMPERATURE
- Sampling temperature (0.0 to 2.0) (default: 0.1)
- --top-p TOP_P Top-p sampling parameter (default: 0.95)
- --seed SEED Random seed (default: current timestamp) (default: None)
- -n, --max-tokens MAX_TOKENS
- Maximum number of tokens to generate (default: 512)
- --stream STREAM Enable streaming output (default: True)
- --echo ECHO Echo the input prompt (default: False)
- --suffix SUFFIX Suffix for fill-in-the-middle request (Codestral)
- (default: None)
-
-Mode Selection:
- -i, --interactive Run in interactive/chat mode (default: False)
- --instruct Run in instruction mode (default) (default: True)
-
-OpenAI-compatible server:
- --server Run the OpenAI-compatible HTTP server instead of inference (default: False)
- --port PORT Server port (default 8080) (default: 8080)
-
-Benchmark (llama-bench style):
- --bench Run the llama-bench-style benchmark (bench.LlamaBench) instead of inference (default: False)
- --bench-args BENCH_ARGS
- Extra benchmark options (use --bench-args="..."), e.g. "-p 512 -n 128 -d 0,4096 -r 5 -o md" (see LlamaBench) (default: )
-
-Hardware Configuration:
- --gpu Enable GPU acceleration (default: False)
- --opencl Force the OpenCL backend when the installed TornadoVM SDK has more than one (default: auto-detected) (default: None)
- --ptx Force the PTX backend when the installed TornadoVM SDK has more than one (default: auto-detected) (default: None)
- --cuda Force the CUDA backend when the installed TornadoVM SDK has more than one (default: auto-detected) (default: None)
- --metal Force the Metal backend when the installed TornadoVM SDK has more than one (default: auto-detected) (default: None)
- --gpu-memory GPU_MEMORY
- GPU memory allocation (default: 14GB)
- --heap-min HEAP_MIN Minimum JVM heap size (default: 20g)
- --heap-max HEAP_MAX Maximum JVM heap size (default: 20g)
-
-Debug and Profiling:
- --debug Enable debug output (default: False)
- --profiler Enable TornadoVM profiler (default: False)
- --profiler-dump-dir PROFILER_DUMP_DIR
- Directory for profiler output (default: None)
-
-TornadoVM Execution Verbose:
- --print-bytecodes Print bytecodes (tornado.print.bytecodes=true) (default: False)
- --print-threads Print thread information (tornado.threadInfo=true) (default: False)
- --print-kernel Print kernel information (tornado.printKernel=true) (default: False)
- --full-dump Enable full debug dump (tornado.fullDebug=true) (default: False)
- --verbose-init Enable timers for TornadoVM initialization (llama.EnableTimingForTornadoVMInit=true) (default: False)
-
-Command Display Options:
- --show-command Display the full Java command that will be executed (default: False)
- --execute-after-show Execute the command after showing it (use with --show-command) (default: False)
-
-Prefill-Decode Optimizations:
- --with-prefill-decode
- Enable single-token prefill decode (default: False)
- --batch-prefill-size N
- Enable batching in prefill when --with-prefill-decode is active and N>1. (default: None)
-
-Advanced CUDA Features:
- --cuda-graphs Enable CUDA graph capture/replay (llama.cudaGraphs=true); PTX backend only. (default: False)
-
-Advanced Options:
- --opencl-flags OPENCL_FLAGS
- OpenCL compiler flags (default: -cl-denorms-are-zero -cl-no-signed-zeros -cl-finite-math-only)
- --max-wait-events MAX_WAIT_EVENTS
- Maximum wait events for TornadoVM event pool (default: 32000)
- --verbose, -v Verbose output (default: False)
-
-```
+## ๐พ GPU memory
-## ๐ OpenAI-compatible server (`--server`)
+Default device allocation is **14GB**. Larger models need more โ raise it with `--gpu-memory`:
-Serve the model behind the HTTP API OpenAI clients already speak โ no external dependencies
-(JDK `HttpServer`), streaming (SSE) and non-streaming.
+| Model size | Recommended | Flag |
+|------------|-------------|------|
+| 1B | 14GB (default) | โ |
+| 3โ7B | 15GB+ | `--gpu-memory 15GB` |
+| 8B+ | 20GB+ | `--gpu-memory 20GB` |
```bash
-llama-tornado --gpu --model model.gguf --server --port 8080
-# or directly:
-java ... org.beehive.gpullama3.server.OpenAIServer --model model.gguf --port 8080 --gpu
+./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-fp16.gguf \
+ --prompt "Tell me a joke" --gpu-memory 15GB
```
-Endpoints: `POST /v1/chat/completions`, `POST /v1/completions`, `GET /v1/models`, `GET /health`.
-
-```bash
-curl http://localhost:8080/
-
-curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
- -d '{"messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":16}'
+Still out of memory? Use Q4_0 instead of Q8_0, or close other GPU apps. The error to look for:
-# streaming
-curl -N http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
- -d '{"messages":[{"role":"user","content":"Tell me a joke"}],"stream":true}'
+```
+TornadoOutOfMemoryException: Unable to allocate ... bytes of memory.
+To increase the maximum device memory, use -Dtornado.device.memory=GB
```
-Any OpenAI SDK works by pointing `base_url` at the server. Generation is serialized on one GPU
-context (requests queue); the reusable core is `server/InferenceService`. Smoke test:
-`scripts/server-smoke-test.sh http://localhost:8080`.
+-----------
-## ๐ Benchmarking (`--bench`, llama-bench style)
+## ๐ง Embed in your own tools
-A [llama-bench](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench)-equivalent
-benchmark for the GPU forward path: a cartesian matrix of tests over one or more models โ
-prompt processing (`pp N`), token generation (`tg N`) and combined (`pg`) โ each repeated
-`-r` times after an untimed warmup, reported as **average tokens/s ยฑ stddev**. Timings cover
-the forward pass only (no tokenization, no sampling), matching llama-bench methodology.
+`--show-command` prints the exact Java + JVM invocation used under the hood, so you can replicate it in IntelliJ, Maven, Gradle, or any launcher:
```bash
-# defaults: -p 512 -n 128 -r 5, markdown output
-llama-tornado --gpu --model model1.gguf --bench
-
-# multiple models, custom matrix, CSV
-llama-tornado --gpu --model model1.gguf --bench \
- --bench-args="-m model2.gguf -p 256,512 -n 64,128 -pg 512,128 -d 0,4096 -r 5 -o csv"
+llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf \
+ --prompt "tell me a joke" --show-command
```
-Benchmark options (`--bench-args="..."` โ use the `=` form): `-m` extra models
-(comma/repeatable), `-p` prompt sizes, `-n` generation lengths, `-pg pp,tg` combined tests,
-`-b` prompt-processing batch size โ when >1, `pp` tokens run through the batched-prefill
-tensor-core (MMA) path, `b` tokens per forward (compute-bound, the llama-bench `-b`);
-generation stays single-token decode. Supported models: Llama / Qwen3 / Mistral (FP16).
-`-d` context depths (untimed KV prefill of `d` positions before each timed test, e.g.
-`tg128@d4096`), `-r` repetitions, `-o md|csv|json|jsonl|sql`, `-oe ` second format to
-stderr, `--delay ` pause between tests, `--no-warmup`.
-
-Example output (RTX 4090, CUDA backend):
-
-| model | quant | size | params | backend | test | t/s |
-| ----- | ----- | ---: | -----: | ------- | ---- | --: |
-| beehive-llama-3.2-1b-instruct-fp16 | FP16 | 2.31 GiB | 1.24 B | TornadoVM CUDA | pp512 | 86.62 ยฑ 2.18 |
-| beehive-llama-3.2-1b-instruct-fp16 | FP16 | 2.31 GiB | 1.24 B | TornadoVM CUDA | tg128 | 101.72 ยฑ 0.43 |
-| Qwen3-1.7B-f16 | FP16 | 3.79 GiB | 2.03 B | TornadoVM CUDA | pp512 | 57.29 ยฑ 0.18 |
-| Qwen3-1.7B-f16 | FP16 | 3.79 GiB | 2.03 B | TornadoVM CUDA | tg128 | 57.71 ยฑ 0.10 |
-
-## Debug & Profiling Options
-View TornadoVM's internal behavior:
-```bash
-# Print thread information during execution
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-threads
-
-# Show bytecode compilation details
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-bytecodes
+
+๐ Full command-line options (llama-tornado --help)
-# Display generated GPU kernel code
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-kernel
+```
+usage: llama-tornado [-h] --model MODEL_PATH [--prompt PROMPT] [-sp SYSTEM_PROMPT]
+ [--temperature TEMPERATURE] [--top-p TOP_P] [--seed SEED] [-n MAX_TOKENS]
+ [--stream STREAM] [--echo ECHO] [-i] [--instruct]
+ [--server] [--port PORT] [--bench] [--bench-args BENCH_ARGS]
+ [--gpu] [--opencl] [--ptx] [--cuda] [--metal]
+ [--gpu-memory GPU_MEMORY] [--heap-min HEAP_MIN] [--heap-max HEAP_MAX]
+ [--debug] [--profiler] [--profiler-dump-dir DIR]
+ [--print-bytecodes] [--print-threads] [--print-kernel] [--full-dump]
+ [--show-command] [--execute-after-show]
+ [--with-prefill-decode] [--batch-prefill-size N] [--cuda-graphs]
+ [--opencl-flags FLAGS] [--max-wait-events N] [--verbose]
+
+LLaMA Configuration: --prompt, -sp/--system-prompt, --temperature (0.0โ2.0, default 0.1),
+ --top-p (default 0.95), --seed, -n/--max-tokens (default 512),
+ --stream (default True), --echo (default False), --suffix (FIM/Codestral)
+Mode Selection: -i/--interactive, --instruct (default)
+OpenAI server: --server (run the HTTP server instead of inference), --port (default 8080)
+Benchmark: --bench (llama-bench-style matrix), --bench-args="..." (see Benchmarking below)
+Hardware: --gpu, --opencl/--ptx/--cuda/--metal (auto-detected; force one of the installed
+ backends), --gpu-memory (default 14GB), --heap-min/--heap-max (default 20g)
+Debug & Profiling: --debug, --profiler, --profiler-dump-dir,
+ --print-bytecodes, --print-threads, --print-kernel, --full-dump, --verbose-init
+Command Display: --show-command, --execute-after-show
+Prefill-Decode: --with-prefill-decode, --batch-prefill-size N (tensor-core MMA batch prefill,
+ FP16 & Q8_0)
+Advanced: --cuda-graphs (PTX backend only), --opencl-flags (default: -cl-denorms-are-zero
+ -cl-no-signed-zeros -cl-finite-math-only), --max-wait-events (default 32000), --verbose/-v
+```
-# Enable full debug output with all details
-./llama-tornado --gpu --model model.gguf --prompt "..." --debug --full-dump
+
-# Combine debug options
-./llama-tornado --gpu --model model.gguf --prompt "..." --print-threads --print-bytecodes --print-kernel
+```bash
+# Peek at what TornadoVM is doing
+./llama-tornado --gpu --model model.gguf --prompt "..." --print-kernel # generated GPU kernel
+./llama-tornado --gpu --model model.gguf --prompt "..." --print-bytecodes # TornadoVM bytecodes
+./llama-tornado --gpu --model model.gguf --prompt "..." --debug --full-dump # everything
```
-## Current Features & Roadmap
+-----------
- - **Support for GGUF format models** with full FP16 and partial support for Q8_0 and Q4_0 quantization.
- - **Instruction-following and chat modes** for various use cases.
- - **Interactive CLI** with `--interactive` and `--instruct` modes.
- - **Automatic backend detection** - `llama-tornado`/`llamaTornado` detect and use whichever backend (OpenCL, PTX, CUDA, or Metal) your installed TornadoVM SDK was built with.
- - **Cross-platform compatibility**:
- - โ
NVIDIA GPUs (OpenCL, PTX & CUDA)
- - โ
Intel GPUs (OpenCL)
- - โ
Apple GPUs (OpenCL)
+## ๐บ๏ธ Features & roadmap
-Click [here](https://github.com/beehive-lab/GPULlama3.java/tree/main/docs/TORNADOVM_TRANSFORMER_OPTIMIZATIONS.md) to view a more detailed list of the transformer optimizations implemented in TornadoVM.
+- โ
**GGUF models** โ full FP16, partial Q8_0 / Q4_0.
+- โ
**Chat, instruction, and interactive** modes (`--interactive`, `--instruct`).
+- โ
**Automatic backend detection** โ `llama-tornado`/`llamaTornado` detect and use whichever backend (OpenCL, PTX, CUDA, or Metal) your installed TornadoVM SDK was built with; override with `--opencl`/`--ptx`/`--cuda`/`--metal`.
+- โ
**Cross-platform**: NVIDIA (OpenCL ยท PTX ยท CUDA), Intel (OpenCL), Apple (OpenCL ยท Metal).
+- โ
**Serving** โ OpenAI-compatible API, llama-bench-style benchmarking, tensor-core (MMA) batch prefill.
+- ๐งฉ **Coming next** โ static batched decode, on-device sampling (preview; see [Serving](#-serving-openai-compatible-preview)).
-Click [here](https://github.com/beehive-lab/GPULlama3.java/tree/main/docs/GPULlama3_ROADMAP.md) to see the roadmap of the project.
+๐ [Transformer optimizations in TornadoVM](docs/TORNADOVM_TRANSFORMER_OPTIMIZATIONS.md) ยท ๐งญ [Project roadmap](docs/GPULlama3_ROADMAP.md)
-----------
-## Acknowledgments
-
-This work is partially funded by the following EU & UKRI grants (most recent first):
+## ๐ Acknowledgments
-- EU Horizon Europe & UKRI [AERO 101092850](https://aero-project.eu/).
-- EU Horizon Europe & UKRI [P2CODE 101093069](https://p2code-project.eu/).
-- EU Horizon Europe & UKRI [ENCRYPT 101070670](https://encrypt-project.eu).
-- EU Horizon Europe & UKRI [TANGO 101070052](https://tango-project.eu).
-
------------
+Partially funded by EU Horizon Europe & UKRI grants (most recent first):
+[AERO 101092850](https://aero-project.eu/) ยท [P2CODE 101093069](https://p2code-project.eu/) ยท [ENCRYPT 101070670](https://encrypt-project.eu) ยท [TANGO 101070052](https://tango-project.eu).
## License
-MIT
+[MIT](LICENSE)