How our critical systems fail, and what we do about it
summary
Nationwide healthcare messaging builds a metadata validator and a certification track after finding zero of the messages validated in a day's live data; a cache removal causes outages fixed in 22 hours to fix and drives regression testing and stricter release controls. Traffic grows to 284 million messages during COVID, infrastructure moves to on‑prem Kubernetes and an event log model with Apache Kafka for safe mirror testing: candidate services consume real streams into isolated topics so changes are verified in production without affecting users.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:10Welcome
- 0:31About us
- 1:11What are we
- 2:07How we got started
- 3:05Messaging
- 5:12Message flow
- 5:45What was the problem
- 6:25The problem
- 7:43The new standard
- 8:03How the traffic works
- 8:52Creating a validator
- 10:25The results
- 11:07Certification track
- 12:10Increased traffic
- 13:33Everything grew
- 14:08Architecture
- 14:28Data normalization
- 22:07Fixing the problem
- 27:34Automation
- 29:59Database
- 31:37Distributed systems
- 33:51Apache Kafka
- 38:08Data flow
- 38:55Testing in production
- 43:37Testing in API world
- 45:13Replaying input
- 48:18Mirror testing
- 50:00Conclusion
- 52:39Questions
related talks
27mObserving Failure in Reactive Systems
41mLessons from Mission-Critical Systems in Fast-Moving Industries
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
16mBuilding Resilient Healthcare Platforms at Scale
32mReliability in Finance: Delivering Quality Market Data at Scale
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
19mLive Q&A
02mOpening Remarks