Five things every developer should know about building mission-critical systems

on YouTube ↗01h 02m

" summary

Five concrete practices for reliable payments: use retries with a unique idempotency key to avoid double-booking, add a circuit breaker to stop endless waits, and decouple with outbox/inbox for guaranteed delivery and surge buffering. Add end-to-end tracing and metrics via OpenTelemetry (one demo trace shows 60 spans) so silent crashes are visible. Scale background workers with leader election and partitioning, and validate with local resilience and load tests using Aspire and K6 (one SOAP bug appears at ~100 concurrent requests).

AI-written summary · about

" chapters (15)

1. 0:00Mission-critical systems intro
2. 2:43Handling server errors
3. 6:10Idempotent services
4. 8:21Circuit breakers for delays
5. 12:39Outbox pattern decoupling
6. 16:54Observability with tracing
7. 21:09Leader election scaling
8. 24:49Inbox pattern buffering
9. 27:41Idempotency demo
10. 31:15Outbox and inbox messaging
11. 34:05Resilience testing automation
12. 38:45Local load testing
13. 44:23AI load testing
14. 51:53Dev containers standard
15. 54:24Five key lessons

" related talks

• 52mPerformance & Stability Anti patternsAlex Mackey · NDC Sydney 2016
• 41mLessons from Mission-Critical Systems in Fast-Moving IndustriesNiek van Raaij · DevDays || DevOps Pro || CyberWiseCon Europe 2025
• 48mFault Tolerance on the Cheap Making Systems That Probably Won't Fall OverBrian L Troutwine · LambdaConf 2015
• 58mSix Little Lines of FailJimmy Bogard · NDC Porto 2019
• 43mGetting Your System into Production and keeping It ThereEoin Woods · GOTO London 2015
• 39mTry Catch Blocks for your Distributed SystemCecilia Deng · REdeploy 2018
• 38mSoft Failures, Hard Goals - Accelerating Payments at Scale During the PandemicKyle Guichard, Venus So · SREcon20 Americas
• 10mBuilding an 'Oops-Less Operation' at ScaleChris Engelbert · JJUG CCC 2019 Fall