Zero to data infrastructure

on YouTube ↗31m

" summary

Explains how streaming with Kafka gives multiplexing, replay and a two-week retention window, recommends managed Kafka and S3 as cheap immutable backups, lays out cloud EDWs like Snowflake and BigQuery and Redshift, contrasts columnar warehouses with row stores for large aggregations, describes a real production failure that caused ~8–9% data loss from a faulty client library, and demands automated QC, monitoring and a culture of shared ownership so wrong metrics get caught before they drive decisions.

AI-written summary · about

" chapters (48)

1. 0:00Introduction
2. 0:40Case Study American Express
3. 0:58Information is the oil of the 21st
4. 1:08Big data is the secret
5. 1:35Value
6. 2:45Data Infrastructure
7. 3:05Construction History
8. 3:20Google Analytics
9. 4:20Build your own solution
10. 4:48Agile way
11. 5:22Why build your own
12. 6:09It cannot be joined
13. 6:32Laws and such
14. 6:53The leading is hard
15. 7:18Take control
16. 7:38Superheros
17. 8:15Technologies
18. 9:12Streaming
19. 9:26Apache Kafka
20. 10:07Producers
21. 11:05Message brokers
22. 11:49Thirdparty dashboard
23. 12:02Permanent storage
24. 12:39Data link
25. 12:49Replay
26. 13:19Buffer
27. 13:56Semantics
28. 14:56Guaranteed ordering
29. 15:48caveats
30. 16:26the hard way
31. 16:56programming languages
32. 17:06cloud enterprise data warehouse
33. 17:50cloud databases
34. 19:25batch processing systems
35. 20:00data center layout
36. 20:28enterprise data warehouses
37. 21:48batch load
38. 23:28quality control
39. 24:03continuous checking
40. 24:37Assume its wrong
41. 25:19Poop goes out
42. 25:36You are the gatekeeper
43. 26:26Monitoring tools
44. 27:19Checks and balances
45. 28:00Agile vs waterfall
46. 28:33Data engineering principles
47. 29:53Book recommendations
48. 30:13Surprise announcement

" related talks

• 39mModern Event-driven Architecture: Adopting Data in MotionAlex Stuart · GOTO EDA Day 2022
• 32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any ScaleGolan Nahum, Ilya Gulman · hayaData 2024
• 58mTales of Data Architecture EvolutionJosef Goldstein · NDC Oslo 2023
• 31mFrom Warehouse to Lakehouse: Data at RestJan Kamieth · bit summit 2025
• 41mStreaming your database - Easier said than done?Sigal Shaharabani · Øredev 2025
• 42mBuilding a Streaming (E-health) Data Pipeline: When and How?Zohreh Jafari · PyData Yerevan 2022
• 25mPlatform Engineering for Modern Data InfrastructureRahul Joshi · Conf42 Platform Engineering 2025
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020