From a hack to the data mesh, the evolution of data engineering...
summary
Leboncoin describes how its data practices evolved from early hacky solutions to a data engineering and data mesh approach to handle massive growth (30 million monthly users). The talk traces operational challenges faced during scaling—legacy problems, unreliable pipelines (“zombies”)—and the technical and organizational changes made to build a more robust data platform.
AI-written summary · about
chapters (14)
- 0:00Introduction and presentation
- 1:15Staging the data stack of Le Bon Coin
- 4:47Objectives of the data stack of Le Bon Coin
- 8:45System eventuality
- 13:37The code
- 16:37The organization
- 18:39The RFC
- 24:27Headcount
- 30:31Kafka topics
- 32:14Advantages of Kafka
- 32:55The data discovery tool
- 33:35The data quality monitoring tool
- 34:33ML pipelines
- 41:51Conclusion
related talks
34mTales of Data Architecture Evolution
38mLesson learned implementing a large-scale data mesh transformation
31mData Mesh In Practice – How To Set Up A Data Driven Organization
51mData Mesh – Successful with People, Process, and Technology
40mData Engineering in a Rapidly Changing World
23mHow To “Data Mesh”: What You Need To Know To Make It Real
43mBoF: Data Mesh Community of Practice: What It Is, Why It's Hard, and How to Help...
25m"Data Engineering: The World Between Worlds"