Data Platform Architecture, Evolution, and Philosophy

on YouTube ↗49m

" summary

Netflix shifts its analytics off HDFS into S3 as central data hub, running Hadoop 2 on EMR with Pig on Tez, Hive, Presto, Teradata (cloud) and Redshift; an event pipeline called Cyril handles ~350 billion events/day, Priam snapshots Cassandra to S3 for ETL, and several OSS tools (Inviso, Lipstick, Aegisthus) support ops. Job submission and multi-cluster isolation go through Genie job submission, eventual-consistency is mitigated by Semper consistent index (DynamoDB-backed), and realtime insight uses Mantis: source–stage–sink with async execution, cooperative backpressure, and elastic scheduling.

AI-written summary · about

" chapters (55)

1. 0:00Intro
2. 1:32Elastic MapReduce
3. 2:14Freedom Responsibility
4. 2:42Architecture
5. 3:03Event Pipeline
6. 3:54Cassandra
7. 4:38priam
8. 5:08Pig
9. 5:58Traditional Data Warehouse
10. 7:01Reporting Tools
11. 9:06S3 Benefits
12. 9:41Genie
13. 13:05Semper
14. 14:39Hadoop
15. 16:46Invis
16. 18:03Cluster Utilization
17. 18:26Big Data API
18. 18:43Tools
19. 19:39Big Data Portal
20. 20:17Philosophy
21. 20:30Netflix Expense Policy
22. 21:09Development and Deployment Flow
23. 21:42DBAs
24. 22:30QA
25. 22:50Upgrading
26. 23:11Acceptance
27. 23:32Safety Nets
28. 23:46Vending Machines
29. 24:11Rules and Processes
30. 24:29Tell vs Ask
31. 24:58Witching Hour
32. 25:30Netflix Culture Deck
33. 27:44Does Netflix work
34. 28:26A simple question
35. 28:37Does it work
36. 29:25Mantis
37. 29:42Why Mantis
38. 30:07Insights vs Business
39. 30:40Insights Requirements
40. 31:42Building Mantis
41. 32:08Designing Mantis for the Cloud
42. 32:21Achieving Low Latency
43. 32:44Maximizing throughput
44. 33:55Holistic approach
45. 34:11Mantas job
46. 34:44Stage
47. 35:02Sink
48. 35:16Job Example
49. 36:44Demo
50. 38:37Monitoring Dimensions
51. 39:00Privacy Data
52. 40:19Data Slice Dice
53. 43:33Hive Pig
54. 44:42Dynamic DB
55. 46:38Use of Storm

" related talks

• 45mModern Data ArchitectureAlexey Grishchenko · JavaDay Kyiv 2015
• 26mBuilding Financial Systems on Eventually Consistent DBsRahul Pilani · Strange Loop 2018
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 45mAccelerated analytics: Building the next-gen data platform for HertzRohit Sinha, Lucy Meyo, Gowtham Ramu · AWS re:Invent 2019
• 34mTales of Data Architecture EvolutionJosef Goldstein · Build Stuff 2022
• 17mDemocratising data analyticsChristian Thiel, Edoardo Gatti · hub.berlin 2022
• 01h 01mData Infrastructure in a Multi-Cloud EnvironmentWouter de Bie · Crunch Conference 2022
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015