Reliable, High Scale Tensorflow Inference....
summary
Twitter deploys a JVM-backed Tensorflow prediction service using Finagle/Thrift and JNI, serving 50+ models at ~20–40M+ requests per second. Engineering choices include JNI byte-array marshalling to avoid expensive object marshaling, client-side backup requests and model warm‑up via replay or customer probes, and CI performance guardrails that run golden models to catch CPU/memory/latency regressions. Automatic restarts, stats dashboards, jhat and jemalloc leak detection, and controlled rolling repairs prove effective at scale.
AI-written summary · about
chapters (4)
- 0:00Introduction to ML at Twitter
- 10:05Scale challenges in inference
- 19:49Reliability challenges and fixes
- 26:21Q&A session
related talks
01h 02mEnabling High Throughput, Low Latency Inference for Your AI Applications
34mWorking with Tensorflow from the JVM #bbuzz
55mLarge Scale Distributed AI on the JVM
42mScaling to Billions of Real-time Recommendations Daily
29mAchieve real-time inference at scale on AWS with Deep Java Library
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
40mScalable machine-learned model serving
33mDealing with the versioning of production-ready models