All the DataOps, all the paradigms
summary
Choices between Hadoop, warehouses, lakes, streams and meshes change ops cost and iteration speed by orders of magnitude: a 2,000-node dependency graph contrasts with one pipeline per person ratios and a 30-person team for ~20 jobs. Offline batch enables automatic backfills and frozen lake versioned partitions; streaming requires event reprocessing (see Confluent recovery guides). Tools shown: Airflow, Luigi time-aware partitions, MapReduce, Docker → Kubernetes, BigQuery/Blob storage. Claim: same productivity as Spotify can be reproduced with small teams.
AI-written summary · about
chapters (11)
- 0:00Introduction and history
- 6:29DataOps paradigms overview
- 9:43Directional data pipelines
- 14:20Batch vs streaming processing
- 19:57Immutability in data engineering
- 24:12Data warehousing vs functional approach
- 31:16Software engineering culture in data
- 34:55Unix philosophy in data systems
- 36:00Consumer-oriented data products
- 40:14Operational efficiency and innovation
- 42:03Q&A session
related talks
54mData engineering & DataOps: Ask Me Anything
28mDevOps in a Data Warehouse: Inside Out
31mMaster Constant Change & Drive Agility With DataOps
32mBig Data Processing : Where do we go next ?
45mModern Data Architecture
02h 05mBig Data and the Data Lake from the Analyst Perspective
42mTales from the Field: The Future is Now
47mData - The Land DevOps Forgot