Hold My Beer - Load Testing. In Production. On Autopilot.

on YouTube ↗33m

" summary

Outbrain runs production load tests automatically for eight years using a homegrown service called Tulatar load test automation Runner that shifts real user traffic across data centers in warm maintain and cool steps and integrates with Slack for interactive commands. Tests increase frequency fourfold to ~8/month, cut ~15 engineer hours per test cycle to nearly zero, and use a traffic lights method for auto-abort/pause based on profit-impacting KPIs. The system stores run metrics for capacity planning and anomaly comparison.

AI-written summary · about

" chapters (29)

1. 0:00Introduction
2. 1:53Outbrain’s scale and operations
3. 3:56Load testing principles
4. 4:38Production vs test environments
5. 7:25Costs and benefits of production tests
6. 8:29Target traffic calculation
7. 10:35Manual testing challenges
8. 11:36Automating with Tulatar Runner
9. 13:21Automated execution and scheduling
10. 14:26Optimal test timing
11. 15:07Health checks and steps
12. 16:10Traffic shifting and warm-up
13. 16:52Maintaining high traffic
14. 17:55Cooling down and risks
15. 18:16Distributed ownership
16. 19:19Traffic light alerts
17. 20:22Direct architecture checks
18. 21:26Slack integration
19. 22:10Automated alert handling example
20. 23:15Automation outcomes
21. 25:23Cultural shift
22. 26:28Empowering teams
23. 27:08Data-driven decisions
24. 28:12Lotus comparison
25. 28:54Hardware planning
26. 29:57Kubernetes capacity
27. 30:40Scaling automation
28. 31:22Handling failures
29. 32:06Conclusion

" related talks

• 47mLoad Testing You Always WantedAnton Moldovan · QA Fest 2019
• 26mLoad testing websites following the steps of a user journeyRaphael Roems · HUSTEF 2023
• 40mAutomated Performance Testing In PreproductionBilly Hoffman · DevOpsDays Seattle 2018
• 42mLoad Testing Is Complicated: A Case Study of NJ Unemployment InsuranceRob Bayliss · Open Source Summit North America 2025
• 28mStress Testing in Production: The New York Times Engineering Survival GuideShesh Patel · Dash 2019
• 50mFrom Load Testing to Reliability Engineering: Making Performance Testing Predict Production BehaviorStephan Mousset · TestMu Conference 2026
• 37mExploratory Load-Testing with AI Tooling - it's Time to Ramp Up!Ole Lensmar · TestCon Europe 2025
• 48mHow to not Crash in Prod, Load Testing Crash CourseStéphane Landelle · Jfokus 2023