Data engineering & DataOps: Ask Me Anything
summary
Built early video conferencing at Google and later a Spotify internal data platform that democratizes analytics, uses Hadoop-era patterns (immutable inputs, functional transforms) and created Luigi workflow orchestration to scale to thousands of pipelines. Current offering runs a jointly operated “data factory” that delivers cleaned outputs as data value as a service. Claim: immutability plus heavy automation lowers cost of experiments; evidence: banks produce ~500 reports/day, Spotify produces a few hundred thousand datasets/day, Google once reported 1.6 billion/day.
AI-written summary · about
chapters (8)
- 0:00Introduction and setup
- 1:55Introducing Lars and his journey
- 8:12When companies need data help
- 18:52What is DataOps?
- 27:41Automation and data engineering
- 33:40Industrializing data operations
- 40:16Practical steps to adopt DataOps
- 49:10Resources and closing thoughts
