Engineering resiliency - how I learned to love an outage
summary
Orchestration platform integrates with 97 integrations (100+ historically) and maps over 4,000 attributes, so vendor errors directly break onboarding decisions; last week there are about 1,300 vendor errors despite a ~99.99% SLA. Production stack uses Elasticsearch/Kibana, Datadog, and BlazeMeter/RunScope for external end‑to‑end tests, a public status page (status.alloy.com), Notion postmortems, Slack alerts and a small Node app that pulls ES. Failover vendors, classification, and playbooks reduce customer impact but intermittent vendor outages still require active incident work.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:43How To Build Resilient Systems on Top of Shaky Foundations
- 4:17Api Integrations
- 12:59Building Blocks
- 13:12Observability
- 14:08Elasticsearch in Kibana for Logs
- 15:42Automated Testing
- 17:37Error Handling
- 30:40How Do You Lock Down Your Code
- 32:01Creating Simpler Developer Experiences
- 33:53Documentation Should Be Public
related talks
25mBuilding a highly scalable system to assure Business Resilience
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
58mResilient by Design
55mDevOps @Scale (Greek Tragedy in 3 Acts)
41mScaling Reliability Engineering with Tools
31mEscaping operations hell
13mBuilding Resilient Emergency Response Platforms
48mDevops@Scale