Data engineering without walls
summary
Argues that data engineering is the practical bridge from models to production and gives concrete tooling and patterns: Hadoop Spark Elasticsearch Kafka Java Scala Python Linux, a recommended Kafka schema registry for backwards‑compatible schemas, and explicit roles (infrastructure specialists, data technicians, data architects). Claims demand for data engineers is skyrocketing and warns that a 25:2 ratio example of data scientists to engineers fails; recommends governance, schema versioning and a track day every two weeks to grow in‑house talent.
AI-written summary · about
chapters (18)
- 0:00Intro
- 2:28Working at Springer Nature @ Berlin
- 4:21Data Engineering, What is this?
- 11:04The era of the monolith aplication
- 11:47Everything changed in the 2000's
- 13:07The rise of data democracy What is
- 14:03The data anarchism
- 15:16From Anarchism to Governance
- 17:02A framework for Data Governance
- 18:20Squads, Chapters, Tribes, .... for Data
- 21:06Have all responsibilities extended?
- 24:26Architecture patterns and tips
- 29:48Prior role for data engineers
- 30:24Distribution by industry
- 31:46Skills based on company size
- 32:43Data Engineering vs Data Scientists (Skills)
- 33:25Hiring and Retaining Data Engineers
- 36:16References
