Data Incidents - When things go wrong
summary
Timeline from cron chaos to a managed pipeline using Airflow and DBT, Redshift, FiveTran, Segment and Tableau and concrete incident rules: detect by alarms and tests, declare severity, appoint an Incident Commander, open a Jira, notify stakeholders in a Slack channel, run SMEs to mitigate, then hold a post incident review with five whys and tracked action items added to sprints. Outcome: measurable trust regained and repeated incidents drop once backlog fixes land, but expect an initial spike in recorded incidents.
AI-written summary · about
chapters (4)
related talks
19mHow to handle incidents at scale
37mIncidents are a new normal
01h 04mIncident Management - Talk the Talk, Walk the Walk
34mIncidents are new normal
37mLearning from Incidents at Scale; Actually Doing Cross-Incident Analysis
31mThe Art of Turning Incidents into Opportunities
39mNavigating the Chaos: A Holistic Approach to Incident Management
34mIncidents Happen