Cluster Serving: Distributed Model Inference using Big Data Streaming
summary
Cluster Serving is an automated, distributed inference system built on Analytics Zoo that wires models into Apache Spark Streaming and Apache Flink pipelines, exposes REST and Python pub/sub APIs, and ships as a ready Docker image. It loads TensorFlow, PyTorch, Caffe, BigDL and OpenVINO models, does multi‑thread optimizations and cluster scheduling, claims zero code change from laptop to cluster, and in one medical imaging case cuts preprocessing + inference from 1–2 hours to minutes while maintaining low latency and high throughput.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:09Big Data & Model Performance
- 2:08Real-World ML/DL Applications A Complex Data Analytics Pipeline.
- 3:20Integrated Big Data Analytics an Seamless Scaling from Laptop to Distributed Big Da
- 5:15Analytics Zoo Unified Data Analytics and Al Platform
- 6:46What's Serving
- 7:17Example of Classical Web Serving
- 8:09Distributed Model Serving
- 8:52Architecture of Main Version of Cluster Se
- 10:06Advantages of Analytics Zoo Cluster Ser
- 12:55Cluster Serving Workflow Overview
- 13:27Very Quick Start
- 15:30API Introductions - Pub-sub
- 16:15Garbage classification on Tianchi Compe
- 17:17Medical Imaging Analysis
related talks
18mCluster Serving: Distributed Model Inference using Apache Flink in Analytics Zoo
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
40mScalable machine-learned model serving
19mBig Data
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
16m[Opening Keynote] Making Distributed Computing Easy
50mData Science in the Enterprise
40mServing Machine Learning Models at Scale Using KServing