Highload — An Unexpected Journey
summary
A real-time ingestion and transformation pipeline was built using Apache Storm, Kafka (0.2→1.1.0), Cassandra, Elasticsearch, Hadoop, Docker, AWS, and Zookeeper/Consul with Nexus for bootstrap caching. The system uses a custom scheduler to separate CPU- vs RAM-heavy Storm topologies, a deploy cache to reduce topology deploy time to seconds, and grows to 500+ machines; auditing creates about 2 TB of audit data (≈7 audit events per business event). Cassandra compaction caused >50% disk usage and node loss and was addressed with a tuned compaction strategy plus blue-green zero-downtime deployment mechanisms.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 2:49Business goals architecture
- 4:12Agent role data processing
- 7:02Apache Storm processing
- 10:51System monitoring audit
- 13:40Sync vs async processing
- 16:53Scaling problems overhead
- 19:01Cluster optimization
- 20:47Faster deploy caching
- 21:30Backup archiving
- 23:14Cassandra compaction issues
- 25:22Hadoop HDFS optimization
- 26:44Audit events processing
- 27:47Elasticsearch query problems
- 29:56Performance algorithm optimization
- 31:59Lock-free algorithms
- 33:50Content bus hardware optimizations
- 37:06Summary future outlook
related talks
42mCloud based low cost low maintenance scalable data platform
41mScaling Observability: From Chaos to Clarity with Open Source Tools
34mData Engineering Architecture
32mAccelerating Humanitarian Relief with Kubernetes [I]
44mAutomating a Big Data Platform with Jenkins
33mData: The Land DevOps Forgot
42mProcessing Millions of Events per Second with Apache Kafka on Kubernetes
42mPlanning To Fail