Managing Real time Data Teams
summary
Moving from batch to real time is a complexity jump, not a lateral change: operations must support minutes-level SLAs instead of days, clusters like Kafka and Flink need people who can run and recover them, and data engineers must be software engineers rather than DBAs. Real-time data products require stable schemas (prefer Avro or protobuf), clear ownership so the business can rely on them, high-bandwidth collaboration between data engineering and data science, and adjusted headcount ratios or consulting to avoid crippling technical debt.
AI-written summary · about
chapters (7)
- 0:00Big Data Institute
- 8:55Business SLAS
- 14:33Data Products
- 18:29Connections
- 20:17Right Ratios
- 26:08Management needs to communicate the value provided
- 28:57Management is responsible for the early success and failures of real-time data teams
related talks
40mData Engineering in a Rapidly Changing World
38mBuilding Effective Data Teams: Strategies for Success
46mData-Driven Transformation: Organizational Changes and Leadership Insights
29mBuilding successful data teams in the realities of 2023
31mFuture of data engineering: what's in store?
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
38mThe Rise & Downfall of the Data Engineer REVISITED
35mBuilding Data-Intensive Teams