Scaling Telemetry Systems with Streaming
summary
Company telemetry grows from 250k to 2.5M events/sec so Kafka becomes the single ingest path; architecture uses ~100 partitions (~200 indexers) with ~50k msgs/sec per partition, producers batch to improve compression, consumers checkpoint hourly and checkpoint hourly to S3 for replay. Operators move from I3 to I4/Graviton, favor local NVMe over EBS, enable follower fetch, use Zstandard compression, run weekly broker/consumer chaos tests, avoid rolling restarts for consumers and measure user-facing freshness SLOs.
AI-written summary · about
chapters (10)
- 0:00Start
- 5:44Streaming data decouples systems
- 18:36Per-producer limits
- 21:47Broker limits
- 24:46Per-consumer limits
- 27:18Consider not running brokers yourself
- 29:24Fully utilise Kafka's replication
- 30:55Use the most efficient base you can
- 34:37Continuously chaos test your DR strategy
- 36:24Application vs system observability
related talks
56mScaling Telemetry Systems with Streaming & Serverless
40mManaging to Your SLO Amidst Chaos
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
30mThe Challenges of Managing Open Source Infrastructure at Bloomberg
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
26mKeynote: 10 Million Spans Per Second: Lessons From Scaling OpenTelemetry at Reddit