A Bug's Life
summary
A low‑latency market data outage traces to a full disk that causes the Airflow nightly ticker job stuck 15 hours (should finish in ~5), discovery via Datadog and Argo CD logs, and a cleanup task that lived in a different repo so Copilot and cron jobs miss it. Fix is a manual wipe, rerun with more workers, plus adding disk pressure alerts at 85%, runbooks for L1, moving cleanup into the main repo, synthetic tests, and prod‑snapshots for non‑prod.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 7:37Developer to engineering manager
- 8:43Attitude toward bugs
- 12:15DORA metrics today
- 17:12Shift left and observability
- 22:54Postmortem fixes
- 25:05Bug's life cycle
- 25:49Q&A
related talks
39mSell cron, buy Airflow: Modern data pipelines in finance
26mThe Sounds of Silence: Lessons from an 18 hour API outage
32mReliability in Finance: Delivering Quality Market Data at Scale
09mLightning Talk: How to Avoid a Kubernetes Doom Loop
34mDatacenter Fires and Other 'Minor' Disasters
33mA Tale of Two System Crashes: Innovation and Accidental Chaos
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
22mOops, I deployed too hard