Cloud based low cost low maintenance scalable data platform
summary
A cloud data platform is built for Myntra using an HTTP-finagle front end, Kafka ingestion, S3 storage and EMR batch processing, with a Storm-based real-time layer in proof of concept. It handles ~20 million events/day (~800M data points, ~25 GB/day), runs ~100 jobs/day (largest job ~2 billion events), was developed by a small team in months and runs at roughly $35/day, achieving scalable fault tolerant pipelines and elastic storage without managing disks.
AI-written summary · about
chapters (12)
- 0:00Start
- 2:00Why hunt elephant to sell shoes?
- 4:59Use case : List products based on CTR
- 6:31Use case : User segmentation
- 8:06Use case : Recommend similar products
- 10:04Constraints
- 12:19Design goals
- 13:58Cloud computing
- 15:51A very basic Big Data system
- 18:38Architecture Diagram
- 26:34Some numbers
- 27:47Some key learnings
related talks
38mHighload — An Unexpected Journey
08mPlatform Engineering at Scale
33mData: The Land DevOps Forgot
12mCloud-Native Platform Engineering
37mBuilding a Data Platform from scratch
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
57mCloud Adoption Patterns
45mResilient Predictive Data Pipelines