Zero to Scale: Telemetry pipeline with Apache Cassandra
summary
Processes send tens of billions of messages a day (≈40 TB, peaks >1M/min) and a telemetry pipeline is built using ZeroMQ proxy to funnel thousands of local producers into Kafka, Kubernetes consumers, and Apache Cassandra storage; messages are aggregated periodically to reduce load and tuned at runtime with feature flags to change publishing frequency, worker counts and writer threads. Hot partitions force a synthetic bucket ID plus date sharding and three cluster-size iterations; after tuning the pipeline reaches target throughput.
AI-written summary · about
chapters (7)
- 0:00Introduction to telemetry project
- 2:09Problem and pipeline overview
- 4:17Development phase and design
- 8:53Storing data with Cassandra
- 10:20Stability and parameter tuning
- 13:32Cassandra optimizations and hot partitions
- 17:05Key takeaways and conclusion
related talks
35mKafka Deployment at Scale
53mDemystifying Kafka
23mOpenTelemetry and Apache Cassandra
18mDesigning Scalable and Extendable Data Pipeline for Call of Duty Games
13mScaling Kafka on Kubernetes
36mTelemetry Analysis for Scaling
31mFun with Sensors
32mMigrating Workloads from Mainframes to Cassandra - Problems, Experience and Learnings