Jlama: A Native Java LLM inference engine
summary
A pure-Java LLM inference engine that runs local Llama/Gemma/Mixtral models on CPU, using the Java 21 Vector API and SafeTensor files, claims GPU-free next-token inference with quantization (block size 32) and batching to cut memory bandwidth and latency. Implements Flash Attention, Mixture-of-Experts and Group Query Attention, uses tiled GEMM to fit L1/L2 caches, exposes an OpenAI-compatible REST layer, faces JVM intrinsic and vector-species tuning issues, and ships as LangChain4j integration with SafeTensor format support.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:51About Jlama project
- 8:59How LLMs work simply
- 14:25CPU vs GPU for LLMs
- 16:57Quantization in LLMs
- 20:30Batching tokens for speed
- 23:48How quantization works
- 27:01Vector API challenges
- 32:01Attention caching explained
- 35:34Sharding models for scaling
- 43:51Practical use cases
- 46:40Future of Jlama
related talks
47mPractical LLM Inference in Modern Java
23mLLMs on the Edge: The Future of On-Device Intelligence
49mTame your Llama, run AI in Java
35mOn‑Device LLMs on Android Devices 2025
53mJava meets AI: Build LLM-Powered Apps with LangChain4j
46mThe Era of AAP: Ai Augmented Programming using only Java
33mllm-d: Kubernetes Native Distributed Inferencing
40mOpen Source LLMs in the Cloud: Scalable Solutions