Auto-detect TornadoVM backend in llama-tornado / llamaTornado

[stratika]

Description

This PR enables the automatic detection of the sdk, so users do not need to use the opencl/--ptx/--cuda/--metal flags from llama-tornado and llamaTornado.

The backend is now auto-detected from the installed TornadoVM SDK ($TORNADOVM_HOME/etc/tornado.backend, which the TornadoVM installer already writes, e.g. tornado.backends=cuda-backend), so the same command works unmodified against any single-backend SDK - 5.1.0-jdk25-opencl, -ptx, -cuda, or -metal.

If an SDK bundles more than one backend, priority order cuda > ptx > opencl > metal picks one. If users need to use a lower priority backend, they can still use the opencl/--ptx/--cuda/--metal flags from llama-tornado and llamaTornado.

Both scripts print which backend they picked, e.g.: Detected TornadoVM backend: cuda (from .../etc/tornado.backend).

Why

Backend selection was a manual flag that had to match whatever SDK happened to be on TORNADOVM_HOME — easy to get out of sync (wrong flag against a single-backend SDK silently produces confusing failures downstream). Since the SDK itself already records which backend(s) it was built with, the launcher can just read that instead of asking the user to repeat it.

The existing flags are becoming optional, and remain as a way that users can explicitly ask for a backend in case an SDK bundles multiple backends.

Compatibility

• Passing an old flag still works as long as it is defined in the used SDK (TORNADOVM_HOME/etc/tornado.backend`).
• llamaTornado (Java) was missing a CUDA backend case entirely (only had OPENCL/PTX/METAL), so it couldn't previously build a working command against a CUDA-backend SDK at all. Added, mirroring llama-tornado.
• Verified against the --bench/--server features merged in from main in the meantime — both still resolve the backend before dispatching to LlamaBench/ OpenAIServer/LlamaApp, no flag collisions. Found and fixed 3 README examples for those features that used the now-removed --cuda flag (would have errored if copy-pasted), and refreshed the stale --help dump.

Where

• llama-tornado, llamaTornado — the actual auto-detection + CUDA support
• .github/workflows/build-and-run.yml, .github/actions/run-inference/action.yml - drop now-redundant --<backend> flags (each CI matrix leg already builds and points TORNADOVM_HOME at a single-backend SDK)
• scripts/benchmark_backends.sh, scripts/all.sh - same
• README.md, .claude/skills/build-n-run-engine/SKILL.md, .claude/agents/gpullama-benchmarking-specialist.md - docs

How to test

1. Point TORNADOVM_HOME at any single-backend TornadoVM SDK (source ./set_paths or equivalent) and confirm $TORNADOVM_HOME/etc/tornado.backend exists.

2. Auto-detection + correct backend wiring:

./llama-tornado --model <path.gguf> --show-command                                                                                                                    
./llamaTornado  --model <path.gguf> --show-command                                              

Both should print Detected TornadoVM backend: (from .../tornado.backend) and the printed java command should reference that backend's exportLists/-exports and tornado.drivers. module.

3. Changed backend flag guard to be optional (for multi-backend bundled SDKs):

./llama-tornado --model <path.gguf> --cuda                                                                                                                               
./llamaTornado  --model <path.gguf> --ptx                          

For example on M4 with OpenCL and Metal:

./llama-tornado --model /opt/models/Llama-3.2-1B-Instruct-F16.gguf --gpu --metal                
Detected TornadoVM backends: opencl, metal (from /Users/thanos/repositories/TornadoVM/dist/tornadovm-5.1.1-jdk25-dev-metal-opencl-mac-aarch64/tornadovm-5.1.1-jdk25-dev-metal-opencl/etc/tornado.backend) - using metal (requested via --metal)

4. New features still compose correctly with detection:

./llama-tornado --model <path.gguf> --bench --bench-args "-p 512 -n 128" --show-command                                                                                  
./llama-tornado --model <path.gguf> --server --port 8080 --show-command                                                                                                  
./llamaTornado  --model <path.gguf> --bench --bench-args "-p 512 -n 128" --show-command         

Command should target bench.LlamaBench / server.OpenAIServer respectively, still with the correct backend's exports/modules.
5. Real end-to-end run (drop --show-command, or add --execute-after-show) against an actual GGUF model, to confirm the JVM actually starts and TornadoVM initializes on the detected backend.
6. If SDKs for more than one backend are available, repeat step 2 against each to confirm detection tracks whichever TORNADOVM_HOME is currently set, with no stale state between runs.

Repo CI (build-and-run.yml) exercises the opencl/ptx/cuda matrix end-to-end on push, which is the main regression check for backend wiring across all three.

Test with SDKs that contain multiple backends

Use the full option from SDKMAN!, or if you do not have SPIR-V build with at least two backends:

sdk install tornadovm 5.1.0-jdk25-full

or

cd TornadoVM
make BACKEND=opencl,cuda sdk && source setvars.sh

Then go to root directory of GPULlama3 and build the project.

./mvnw clean install
./llama-tornado --gpu --model /opt/models/Llama-3.2-1B-Instruct-Q8_0.gguf 

and it dumps a message when multiple backends are available, the system runs on the highest priority backend which is defined as:

./llama-tornado --gpu --model /opt/models/Llama-3.2-1B-Instruct-Q8_0.gguf
Detected TornadoVM backends: cuda, opencl (from /home/thanos/repositories/TornadoVM/dist/tornadovm-5.1.1-jdk25-dev-cuda-opencl-linux-amd64/tornadovm-5.1.1-jdk25-dev-cuda-opencl/etc/tornado.backend) - using cuda (auto-selected)

To explicitly use the CUDA backend

./llama-tornado --gpu --model /opt/models/Llama-3.2-1B-Instruct-Q8_0.gguf
Detected TornadoVM backends: cuda, opencl (from /home/thanos/repositories/TornadoVM/dist/tornadovm-5.1.1-jdk25-dev-cuda-opencl-linux-amd64/tornadovm-5.1.1-jdk25-dev-cuda-opencl/etc/tornado.backend) - using cuda

To explicitly use the OpenCL backend

./llama-tornado --gpu --opencl --model /opt/models/Llama-3.2-1B-Instruct-Q8_0.gguf
Detected TornadoVM backends: cuda, opencl (from /home/thanos/repositories/TornadoVM/dist/tornadovm-5.1.1-jdk25-dev-cuda-opencl-linux-amd64/tornadovm-5.1.1-jdk25-dev-cuda-opencl/etc/tornado.backend) - using opencl (requested via --opencl)

However, there is the option to use the flag --opencl, --metal, --cuda,--ptx to explicitly ask to run with this backend.

[orionpapadakis]

LGTM