Smart Streams: A Real-time framework for scoring Big and Fast Data
summary
Claims a working real-time scoring pipeline that ingests device CSV/JSON from S3, runs ETL and windowed aggregation in Spark Structured Streaming, calls a deployed inference endpoint as model-as-a-service, registers stream results as queryable tables, and writes scored output to sinks for downstream consumers. Architecture uses loosely coupled microservices and containerized deployment, supports iterative model updates without stopping the stream, and demonstrates an end-to-end activity classifier producing live, SQL-queryable results.
AI-written summary · about
chapters (8)
- 0:00Smart Streams intro
- 3:33ML fundamentals & training
- 8:05Microservices benefits
- 12:42Spark for real-time
- 16:44ML on streaming challenges
- 17:34Demo: activity classification
- 25:48Aggregating user activity
- 29:28Conclusion & Q&A
related talks
37mBig Data Analytics, Machine Learning, Real Time Processing
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
01h 21mStructured Streaming with PySpark and Azure Databricks
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
01h 47mRadanalytics.io streaming applications workshop
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
38mReal Time ML Pipelines in Multi-Tenant Environments
01h 25mApache Spark for Big Data Processing