Architecting for Speed & Scale
summary
Argues for a data integration hub that uses an in-memory data grid for real-time inference and SSD/iceberg archives for long term reconciliation rather than streaming-everything to disk. Cites Trade Desk scale: 10 million auctions/sec with a 20 ms SLA, Vertica for model training and Aerospike for serving, and a real case where 278-node Spark cost $30M/month vs nine-node alternative $1.2M/month. Warns autoscaling and cloud egress can explode bills and notes ~80% hybrid or cloud repatriation trends.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 3:54Machine learning over generative AI
- 10:21Scale up vs scale out
- 18:17Cloud cost and reality
- 28:52Performance vs practicality
- 33:13Trade Desk’s real-time architecture
- 39:46Data integration hub solution
- 41:10Future-proofing architecture
related talks
55mThe Great Data Debate 2022
50mDesigning a scalable data platform based on Apache Spark/Iceberg
38mHow to run Real Time processing on Big Data
40mArchitecting for Data-Intensive Workloads
30mAn architecture for big data: Supporting journalism through transparency in advertising
08mReal-Time Data Systems: Speed, Scale, and Resilience
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure