Prepare Your System To Scale OR Why Auto Scaling Is Not Enough
summary
Oribi handles 100M→300M events/day and survives a sudden 10x spike in one hour by combining Kafka, Elasticsearch, Apache Druid and Kubernetes with specific playbooks instead of relying on autoscaling alone. Key practices are elasticity, batch, isolation: pre-scale Kafka before rebalance becomes impossible, move hot writes to batched ingestion to avoid IO collapse, and isolate heavy dashboard queries. Observability centers on an end-to-end processing time metric, keep headroom (80–85%) for warmups, and replace unstable spot instances for critical stateful stores.
AI-written summary · about
chapters (8)
- 0:00Introduction and welcome
- 1:08Why auto scaling isn't enough
- 5:52Three pillars for scaling
- 7:39Elasticity as a mindset
- 11:13Batch processing benefits
- 13:43Avoiding starvation with isolation
- 15:56The role of observability
- 20:53Q&A on JVM and scaling costs
related talks
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
34mAutoscaling Made Easy with Kubernetes Event-Driven Autoscaling (KEDA)
09mKeynote: Scaling Observability Pipelines in the Enterprise
40mSurvivability of Observability: Taking your DIY system to success
41mScaling Observability: From Chaos to Clarity with Open Source Tools
27mObserving Failure in Reactive Systems
26mGrowing Pain: Scaling to 10K Workflows per Week
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale