Stabilizing + Reinforcing H-E-Bs Curbside Fulfillment Systems While Reinventing Them
summary
H-E-B stabilizes a brittle curbside monolith while building microservices by inserting anti-corruption services as buffers, running weekly game days with Gremlin chaos engineering, and adopting Linkerd/service mesh on Kubernetes. DB work fixes 16 bad queries (one was 7% CPU) and cuts query latency from seconds to milliseconds, improving app performance >30% and reaching four nines uptime. Three-day VM scaling and 3–5x end-to-end load tests double throughput with no production degradation.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:26Background
- 3:20Incidents
- 3:55The Impact
- 6:48DataDriven Approach
- 8:53Chaos Engineering
- 9:44Databases
- 10:56Performance
- 14:48Chaos Work
- 16:10Buoyant
- 16:57Chaos Tests
- 17:15Red vs Blue
- 18:44Coop Doom
- 19:18First Store
- 19:29Moving Faster
- 19:43Impact on Customers
- 20:02Impact on Engineers
- 20:41Final Thoughts
- 21:51Questions
related talks
36mShifting Performance Engineering Left
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
42mHow Houghton Mifflin Harcourt Went from Months to Minutes for Infrastructure Delivery
01h 02mTools for simplified large scale systems
30mThe Bitter and the Sweet of Running a Planet-Scale Build &...
29mThe Road to Reliability
35mLiving in a Hybrid World
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months