Enabling High Throughput, Low Latency Inference for Your AI Applications
summary
Demonstrates a hybrid architecture that runs lightweight models inside the JVM for millisecond inference and pushes computation into an in‑memory data grid when data locality is required. Uses Spring Boot + Spring Data for GemFire with VMware Tanzu GemFire as model store/cache and compute grid, and runs exported models via ONNX runtime in JVM. Demo shows local inference from ~0.74s to ~0.07s after caching and a distributed review-count function ~1.66s vs ~0.02s on cache hits, with a model loader that propagates updates.
AI-written summary · about
chapters (8)
- 0:00Introduction and setup
- 1:11Inference in modern enterprise
- 5:47Edge and data local patterns
- 9:37Demo tech stack overview
- 15:02Edge inference demo
- 32:59Data local inference demo
- 46:21Gemfire features deep dive
- 55:12Key takeaways and Q&A
related talks
55mLarge Scale Distributed AI on the JVM
55mPetabyte-Scale AI Memory with Serverless Java
29mReliable, High Scale Tensorflow Inference....
29mAchieve real-time inference at scale on AWS with Deep Java Library
51mIntegrate Machine Learning into Your Spring Application in Less than an Hour
30mGo, ONNX and AI in Production
17mGreen AI with JobRunr: Sustainable training and inference on the JVM
27mAI inference on the Edge cloud using WebAssembly