Panel Debate: Rise Of The Data Engineer
summary
Panel argues data engineering now means cleaning decades of technical debt and plumbing lots of varied sources into usable products, with hiring demand described as a nightmare and Spark usage shifting toward ~45% Python and ~42% SQL. Teams build reproducible lakes (lakeFS / lakehouses), add quality gates (Great Expectations) and measure ROI, while daily BAU can consume 10–50% of time. Core skills center on SQL + Python, orchestration, cost-aware design and domain-aware modeling.
AI-written summary · about
chapters (8)
- 0:00Introduction to panel
- 1:45What is a data engineer
- 7:26Why data engineering is popular
- 10:18Data as a product
- 18:05How to get into data engineering
- 28:56What data engineers should not do
- 36:18Biggest trends in data engineering
- 47:01Why data engineering is exciting
related talks
31mFuture of data engineering: what's in store?
50mThe future of Data Engineering panel
38mThe Rise & Downfall of the Data Engineer REVISITED
40mData Engineering in a Rapidly Changing World
22mDataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi
28mDataEngBytes Panel discussion - Sydney
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
28mSoftware Engineering Skills for Data Engineers