Lessons Learned Implementing a Resilient, Distributed, Real-Time Architecture
summary
A Teads engineer describes building a resilient, distributed real-time accounting system that tracks 50,000 budget counters with peak write throughput >60k writes/s and read throughput ~1.5M reads/s to stop ad delivery precisely. The talk covers the architecture and operational choices around stream vs batch processing, delivery guarantees (exactly-once vs at-least-once), ACID vs CAP trade-offs, database architecture, and the Lambda architecture pattern.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 4:31Distributed counter problem
- 11:55System requirements
- 13:31Naive counter approaches
- 16:05Read/write architectures
- 20:46Cassandra choice and limits
- 21:50Upstream event handling
- 24:01Kafka and Flink pipeline
- 27:16Idempotence and processing
- 31:19Error correction and Lambda
- 36:23Architecture summary
- 37:08Q&A
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation
25mMixing data streams at scale
44mHow to Handle Over 1 Trillion HTTP Requests/day: Behind the Scenes of the Ad Tech Industry
49mWhen Performance, Scalability and Robustness Upset Your Architecture
41mTracing Real-Time Distributed Systems
32mReal time customer experience across industries
44mAdvertising at the Speed of Light (DMP Platform)