From Load Testing to Reliability Engineering: Making Performance Testing Predict Production Behavior

on YouTube ↗50m

" summary

Performance testing only becomes predictive when it integrates objectives, observability and production feedback rather than producing reports; at scale — 350,000 deployments a week — manual graph interpretation fails. Define SLIs and SLOs, track the four golden signals (latency, traffic, errors, saturation), choose tests by risk (six test types: load, stress, spike, volume, endurance, resilience), build workloads with a six-step workload method, validate against production, and beware misleading results like perfect scripts or insufficient VMs discovered during DR and spike experiments.

AI-written summary · about

" chapters (51)

1. 0:00From Load Testing to Reliability Engineering
2. 1:35Speaker Intro - Stephan Mousset, Global Performance and Resilience Platform, ING
3. 2:07Performance Testing Only Becomes Predictive When It Stops Being Isolated
4. 2:38It's an Engineering Feedback Problem, Not a Testing Problem
5. 3:40The Valuable Output Is the Feedback, Not the Report
6. 5:44Scale at ING: 350,000 Deployments a Week
7. 6:45Why Interpreting Every Graph Manually Doesn't Scale
8. 7:16Load Testing Produces Data; Reliability Engineering Produces Decisions
9. 7:48The Iceberg: What Sits Below the Waterline
10. 8:51Performance Testing Is a System, Not a Script
11. 9:53The Connected Considerations: Workloads, Objectives, Observability, Data
12. 11:59Not a Quality Gate at the End - Continuous Feedback
13. 13:01You Cannot Test What You Have Not Defined
14. 13:31Starting From the Intended Outcome, Not the Script
15. 14:31The Four Golden Signals: Latency, Traffic, Errors, Saturation
16. 16:36Latency, and Why Slower Sites Make Less Money
17. 17:40Errors and What Saturation Actually Tells You
18. 18:42Balancing Resources So You Can Reach a Real Limit
19. 20:16Bottleneck Hunting and the Reliability Chain
20. 20:48SLIs vs SLOs vs Golden Signals
21. 21:18Why Saturation Makes a Poor Indicator but a Good Alert
22. 23:23Don't Choose a Test by Habit - Choose It by Risk
23. 24:26Which Failures Do You Actually Need to Understand?
24. 24:58Six Test Types and the Question Each One Answers
25. 25:30Load Test, Stress Test and Finding the Break
26. 26:32Spike Testing and the Ticket-Release Problem
27. 27:04Volume and Endurance Testing - and Why Endurance Is Overrated
28. 27:35A Concrete Example: Concert Tickets Released at 10 AM Friday
29. 29:08Regulators Who Require Long-Duration Proof
30. 29:38Scalability Testing: Does Cloud Scaling Actually Recover Latency?
31. 30:10Resilience Testing: Chaos, Failover and Resource Exhaustion
32. 31:12Disaster Recovery Across Two Data Centres
33. 31:44The Real Finding: Not Enough VMs for High-Load Tests on One Site
34. 32:16A Perfect Script With an Unrealistic Workload Tells You Nothing
35. 33:18Workload Modelling: Turning Production Behaviour Into a Hypothesis
36. 33:48The Six-Step Method, Ending in Validation
37. 35:23Simplification Is Inevitable - Score How Close You Are
38. 35:58What Do the Results Actually Mean?
39. 36:29Correlating Test Behaviour With Application and Infrastructure Behaviour
40. 37:00The Test Result Is the Beginning of the Investigation
41. 37:31Where Automation Removes Toil Without Replacing Judgement
42. 38:02Automating Performance Analysis: Better or Worse Than Before?
43. 39:38Comparing Against Previous Runs and High-Level Targets
44. 40:09Validating Coverage Against What Production Actually Does
45. 41:11Only Production Tells You Whether the Prediction Was Credible
46. 42:14The Feedback Loop That Turns Testing Into Reliability Engineering
47. 43:14Five Practical Things to Start Applying
48. 43:44Make Production Your Friend
49. 44:14Let SLOs and SLIs Drive the Decision, Not Habit
50. 45:16Final Thoughts: Reliability Is How Decisions Are Made
51. 46:48"Test in Prod or Live a Lie" - What Charity Majors Actually Meant

" related talks

• 01h 00mPerformance In Modern Times: An Evolutionary JourneyLeandro Melendez · PNSQC 2022
• 44mPerformance Testing: the way to make a true loadIzzet Mustafaiev · JavaDay Kyiv 2015
• 47mCultivating Production ExcellenceLiz Fong Jones · NDC DevOps 2020
• 45mPerformance testing from scratchRuslan Ziganshin · HighLoad++ Siberia 2018
• 38mImplementing SRE practices: SLI/SLO deep diveDavid N. Blank-Edelman · DevOpsDays Tel Aviv 2018
• 29mProgressive Testing to Meet the Performance ImperativePaul Bruce · DevOpsDays Boston 2018
• 48mHow to not Crash in Prod, Load Testing Crash CourseStéphane Landelle · Jfokus 2023
• 38mPerformance 101Martin Chaov · HackConf 2021 Online edition