What could possibly go wrong?
summary
External vendor delivers a year‑old shipment app that needs 18 months of hands‑on work to stop frequent outages: adding acceptance tests, cleaning logs, adding APM and structured logs, and rewriting the worst code yields two weeks without incidents. A separate Lambda‑based report service (Sauron) integrates shipping via an event bus with lambdas; choosing events introduces duplicated logic, missing tests, incorrect numbers and long delays. Six months after release the feature gets no users, proving the costly event approach didn’t pay off despite good observability and fast deploys.
AI-written summary · about
chapters (12)
- 0:00Introduction and personal story
- 3:49Ship It failure and chaos
- 7:23Stabilizing Ship It over a year
- 10:16Introducing Sauron application
- 12:04Choosing event-driven solution
- 16:52Infrastructure and permission issues
- 17:37Duplicated logic and testing gaps
- 19:22Big-bang approach and its consequences
- 20:50What went right despite struggles
- 23:44Six months later and lessons learned
- 25:38Key learnings and closing remarks
- 27:27Q&A session
related talks
31mOn the Importance of Visibility
42mCatastrophes and Atrophies
40m4 Key takeaways from running Serverless on production for 4 years
01h 02mServerless in production, an experience report
44mServerless Architecture on Production: Good, Bad and Ugly Parts
25mHow to Fail at Serverless (without even trying)
39mServerless in production, an experience report
15mWhy I Prefer Good Testing Over Excellent Testing