Scaling Telemetry Systems with Streaming & Serverless

on YouTube ↗56m

" summary

Telemetry ingestion grows from ~200k to 4 million trace spans/sec so Kafka becomes the persistent, ordered buffer with hourly consumer checkpoints to S3, partitioning by trace ID and using local NVMe on i-class instances to avoid cloud block throttling. Reads run a serverless map phase against S3 with 40,000 concurrent Lambda workers (peaks to 70–80k), 90% of queries return in ~2.5s, retries for the slow 1%, and Graviton reduces Lambda cost ~20–30%.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 3:38Streaming data systems
3. 17:22Kafka cluster performance
4. 26:35Streaming observability
5. 34:01Storage and query challenges
6. 41:45Serverless big data
7. 50:30Streaming vs serverless
8. 51:16Q&A

" related talks

• 41mScaling Telemetry Systems with StreamingLiz Fong Jones, Terra Field · SREcon23 Americas
• 49mUsing serverless functions for real-time observabilityLiz Fong Jones · CPH DevFest 2024
• 30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in ProductionTapasweni Pathak · LambdaConf 2019
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022
• 40mManaging to Your SLO Amidst ChaosLiz Fong Jones · YOW! Melbourne 2022
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 32mBuilding and scaling a log analytics platform - a serverless approachNarendran · Rootconf Cloud Server Management Miniconf Chennai 2017
• 47mServerless in production, an experience reportYan Cui · Build Stuff 2017