How We Achieved 100x Scale
summary
Rebuilt a production unwanted-mail pipeline in two months with two engineers, replacing Python + Celery to support >100x scale at ~10% of prior cost and handle millions of requests; moves from minutes to seconds of processing time. Architecture forwards events into Kafka and uses Golang + C-based Kafka library consumers that spawn ~1000 goroutines per consumer so each instance runs in <100MB and <0.1 vCPU while avoiding partition explosion.
AI-written summary · about
chapters (4)
- 0:00Introduction to the team
- 0:45Scaling challenges
- 2:17Technical implementation
- 4:42Results and conclusion
related talks
28mManaging and Scaling Real Time Data Pipelines using Go
53mGo at Mainframe Scale
39mBuilding Scalable Orchestration
57mLessons in Building a System that Processes More than 70 Billion Events Daily
27mScaling using Go
05mGo for Real Time Streaming Architectures
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
22mOne Billion Row Challenge in Golang