Practical LLM Inference in Modern Java
summary
Shows a working local LLM inference engine written in a pure Java single-file engine (≈2,200 lines) that runs Llama 2/3 and other open models on CPU using Java 21+ features, the Vector API and GraalVM native-image. Demonstrates 32–48 tokens/sec on a 1B Llama 3.2 (earlier 7B run was ~1.5 tokens/sec), explains quantization tradeoffs (16-bit ~3.1 t/s, 8.5-bit ≈2×, 4.5-bit ≈3.5×) and human comfort at 5–8 tokens/sec. Output artifacts are tiny — 73 KB jar and 23 MB native exe — and the Java build sometimes outperforms llama.cpp while integrating with LangChain4j.
AI-written summary · about
chapters (33)
- 0:00Introduction to project
- 0:28Speaker background
- 1:01Presentation overview
- 2:13Project history and origin
- 3:36Llama 3 Java project
- 4:46Small language models
- 6:19LLMs everywhere
- 7:55Initial chat demo
- 9:22Benefits of local Java
- 10:23LLM engine components
- 12:56Transformer architecture
- 15:27LLM weights explained
- 16:08Inference cost analysis
- 17:32Memory bandwidth importance
- 19:15Performance bottlenecks
- 20:10Java code implementation
- 21:59Code size and complexity
- 22:59Model quantization
- 25:14Performance metrics
- 27:30Why Java and GraalVM
- 28:34Vector API in GraalVM
- 30:14JIT and AOT performance
- 31:10Native Image advantages
- 32:12Native Image demo
- 34:09Native executable chat
- 35:10AOT model pre-loading
- 38:31LangChain4j integration
- 40:41Benchmark results
- 42:53Apples to apples comparison
- 44:27JVM outperforming native
- 45:47Supporting other models
- 47:38Future work and GPU
- 49:22Conclusion and Q&A
related talks
47mPractical LLM Inference in Modern Java
49mTame your Llama, run AI in Java
16mHow fast are LLM inference engines anyway?
46mThe Era of AAP: Ai Augmented Programming using Modern Java
47mJVM in the Age of AI: Babylon, Valhalla, TornadoVM and friends
46mThe Era of AAP: Ai Augmented Programming using only Java
24mLLM Inference Optimization From Simple To Sophisticated
11mllama.cpp - faster and easier for everyone!