Infrastructure scale issues
summary
Early signs: pipelines silently fail, dashboards go stale, queries slow, backlog velocity collapses and a full-time hire is loading the initial data. Concrete fixes: add monitoring and alerts, pair engineers on incidents to cut diagnosis time ~4x, run an MVP “slow app” to win exec buy‑in, and prefer managed services when onboarding speed matters — e.g. Databricks over EMR for fast onboarding while using dbt to centralize lineage and reduce ad‑hoc SQL.
AI-written summary · about
chapters (5)
- 0:00Introduction and panel setup
- 2:11Early signs of scaling issues
- 10:59Build versus buy decision
- 19:48Getting leadership buy-in
- 26:29Closing thoughts and summary
related talks
07mFrom Scaling APIs to Scaling Data to Scaling Your Team
38mKeeping pace with data infrastructure so it doesn't outrun you
25mData Demons: Unleashing Chaos or Driving Performance?
20mScale-up your job satisfaction, not your software
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022
45mResilient Predictive Data Pipelines
28mHow to help BI & AI people as a software engineer?