Reliability Lessons from Running High Traffic Services for More Than 10 Years
summary
Runs Canada’s biggest media and retail sites on AWS, handling petabytes of video and Black Friday spikes, and reports specific incidents and fixes: a live voting event grows 3 to 30 servers in minutes, a one-line bug in token/date logic causes a four-hour outage, automation once wiped microservice configs, and an IPv6 change breaks cloud functions globally. Practices that actually move uptime: 70/20/10 test pyramid, game days and chaos experiments (Gremlin/chaos tools), dashboards that drill down to customer-facing metrics.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:25Who am I
- 1:32Slalom
- 2:10Big Brother Canada
- 2:23Testing Inappropriately
- 4:35Black Friday
- 6:05A Peak
- 7:43Multiple Availability Zones
- 9:18Transparency
- 11:50Collaboration
- 13:45Measuring
- 14:48Reliability is team sport
- 15:30I am the boss
- 16:45The test pyramid
- 17:53Keep it simple
- 18:48Delivery doesnt matter
- 19:30The projects will get delivered
- 19:47DevOps
- 22:49Reliability Engineering
- 25:06Cows Engineering
related talks
31mDevOps in the Cloud: Case Studies of Amazon.com teams and their resilient architectures
42mPlanning To Fail
56mGetting all the 99.99(9) you always wanted
25m"The website's down!" Stories and lessons on keeping your website up.
01h 08mPerformance testing in the cloud
52mPerformance & Stability Anti patterns
34mLessons Learned from Five Years of Multi-Cloud
46mManaging a Billion Transactions with Tech, Tactics, and Teamwork