Practical LLM Inference in Modern Java
summary
Single-file Java engine implements local Llama 2/3 inference (2,500 line l3.java) with tokenizers, GGUF parser and Q8/Q4 and FP16/BF16 quantizations, uses Panama memory segments and the Vector API, and targets Java 21+ and GraalVM native-image. Benchmarks show CPU-only runs hitting around 40 tokens/sec (authors aim for 5–8 as minimally acceptable), native-image builds produce a ~23 MB binary and can achieve zero-ms startup when model metadata is pre-baked, and performance sits in the same ballpark as llama.cpp/ollama while remaining pure Java and integrable with LangChain4j.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 1:08Project overview
- 2:06History and technical scope
- 4:58LLMs in the modern ecosystem
- 8:27First project demo
- 11:04Benefits of local Java inference
- 13:51Core architecture and mechanics
- 18:50Memory bandwidth and performance
- 23:24Quantization and speed targets
- 29:22Optimizing with GraalVM
- 33:43Advanced demos and benchmarks
- 39:42Conclusion and future outlook
related talks
51mPractical LLM Inference in Modern Java
51mJlama: A Native Java LLM inference engine
47mJVM in the Age of AI: Babylon, Valhalla, TornadoVM and friends
46mThe Era of AAP: Ai Augmented Programming using only Java
16mHow fast are LLM inference engines anyway?
46mThe Era of AAP: Ai Augmented Programming using Modern Java
49mTame your Llama, run AI in Java
38mThe Era of AAP: Ai Augmented Programming using only Java