Real-Time Data Systems: Speed, Scale, and Resilience
summary
Real-time pipeline targets 180 gigabytes with ~30% realtime demand and latency goals under 1 second, using event capture at ~10 ms, ML scoring at ~50 ms and 80–120 ms end-to-end for fraud blocking. Architecture uses a distributed log for ingestion, Spark Structured Streaming or stateful stream engines for joins and exactly-once semantics, ClickHouse for subsecond analytics on high-cardinality data, and efficient formats like Arrow / protobuff while minimizing network hops and serialization (20–30% cost).
AI-written summary · about
chapters (11)
- 0:00Introduction to Real-Time Data Systems
- 0:36Applications of Real-Time Data
- 1:02Real-Time Data Pipeline
- 1:46Ingestion Layers and Stream Processing
- 2:25Case Study: Fraud Detection
- 3:09Fast Query Engines
- 3:53Engineering for Low Latency
- 4:53Scaling and Consistency
- 6:32Frontend Integration
- 7:16Key Takeaways
- 8:02Conclusion
related talks
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
30mFive Challenges in Real-time Stream Processing and Five Solutions
09mLet's Get Real: Acting on Data in Real Time
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
27mArchitecting Sub-Second Analytics Pipelines
20mBuilding realtime efficient queries in an IOT platform
28mBuilding a Coud Native Data Pipeline for Your Real-Time ML & AI Apps