The overnight failure
summary
A carpool app runs a weekly job that enqueues passenger driver charge jobs into a queue and the payment gateway charges and pays drivers. A bug enqueues thousands of duplicated charge jobs so users get billed repeatedly, with one user charged over $5,000 instead of $50 and many cards maxed out. The team stops the generator, reverses driver transfers, issues refunds and expedited reimbursements because refunds take up to five business days, adds tests and idempotency checks, deploys a fix.
AI-written summary · about
chapters (27)
- 0:00The overnight Failure
- 1:34NARCOS
- 3:01Selfie time!
- 3:42We all have broken the internet
- 3:57We normally don't talk about it in public
- 4:45Impostor syndrome
- 6:55What's the worst thing that could happen to you at work?
- 7:15How?
- 10:14What?
- 10:42Black Saturday
- 13:22Lots of duplicated charges
- 14:16Refunded charges
- 14:42Reversed transfers
- 16:05Wrote tests, a fix and deployed it
- 18:36Refunds take up- to 5 business days
- 18:50Reached out to users to offer an expedited reimbursement option
- 19:28Embarrassing things happen
- 19:40Tests won't save you
- 19:51QA won't save you
- 20:12We need to make admiting mistakes easy
- 20:30Trust that we won't be judged
- 20:55Make sure you understand what happened
- 22:39Document the lesson learnt
- 22:49Don't git blame
- 23:22You are not your failures
- 23:51It's all temporary
- 25:41Recording and Production Confreaks
