The State of Data Engineering
summary
Claims that the industry sees an 85% failure rate for big data projects and declining average value creation over time drive a diagnostic: parallels with failed data warehousing and data science efforts trace to skill gaps, vendor hype, and bad org choices. Specifics include data scientists being roughly 80% inefficient at engineering tasks, about 25% not using Gen at all, overreliance on SQL-only skillsets, misguided tech choices (point-to-point vs pub/sub), and poor business communication; fix by hiring software/distributed-systems engineers, quantify business value and effort, and mentor.
AI-written summary · about
chapters (14)
- 0:00State of data engineering
- 1:09Industry failures and costs
- 3:19Data warehousing history
- 7:19Rise and fall of data science
- 10:32Why data scientists struggle
- 12:40Resume-driven skill inflation
- 15:16Data engineering as new sexiest job
- 17:45Why SQL alone is insufficient
- 20:15Generative AI limitations
- 22:47Focus on unsolved problems
- 24:13Inefficiency in data science teams
- 25:18Importance of optionality
- 27:02Direct business communication
- 30:29Three key business questions
related talks
22mWhy Most Data Projects Fail & How to Avoid It
51mPanel Debate: Rise Of The Data Engineer
40mData Engineering in a Rapidly Changing World
22mDataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi
30mValue Engineering: How to Get Your Big Data Investment.....
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
59mThe Great Data Debate
31mFuture of data engineering: what's in store?