How to Leverage the Apache Kafka Ecosystem to...
summary
Shows a concrete pipeline that trains models with TensorFlow or H2O, exports them as Java artifacts, and embeds them in a single-class Kafka Streams application for real-time inference; a demo predicts flight delays in a few microseconds to milliseconds and prints results to a Kafka output topic. Uses Kafka Connect for ingestion, KSQL for analytics, and schema registry for evolving schemas. Cites LinkedIn at 4–5 trillion messages/day and Netflix at ~6 PB/day to justify scale and reports the microservice approach scales to hundreds or thousands of predictions/sec.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:30What Is Extreme Scale
- 1:24Agenda
- 2:11Why Do People Want To Use Machine Learning Processes
- 7:05Predictive Maintenance
- 8:10Machine Learning Framework
- 10:32What Is Apache Kafka
- 11:46Linkedin
- 13:18Uber
- 21:08Kafka Connect
- 21:31Data Processing
- 22:40Confluent Schema Registry
- 25:49Kafka Perspective of the Open Source Ecosystem
- 28:35Demo
- 39:16Automation and Devops
- 40:12Continuous Batch Updating
related talks
40mHow to leverage the Apache Kafka Ecosystem to productionize Machine Learning
57mMachine Learning Data Pipelines
52mMachine Learning Data Pipelines
19mAI Workflows with Kafka and Modern Architectures
01h 02mBig Data Analytics in Near-Real-Time with Apache Kafka Streams
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
41mStreams to the River: Powering an Online XGBoost Classifier with Kafka
34mActionable Insights with Real-time Streaming Analytics of Customer Reviews