From Machine Learning Startup to Big Data Company
summary
Startup builds an RTB engine that grows from ~500 requests/sec and ~10 GB/day to 10x per stage while keeping 100 ms round trips, and learns the hard way: ingesting JSON logs into MySQL causes tiny files, duplicated data, and slow ETL; homebuilt Java MapReduce is costly to maintain. Fixes that work: use Kafka to decouple ingestion, migrate raw logs to Parquet with a schema (protobuf/Avro), move analytics to Hive/Spark with Luigi pipelines, and provide fast SQL access to raw data so data scientists can iterate.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:30What does it work
- 7:20Bootstrapping
- 9:15Structure
- 11:35Realtime Bidding
- 16:30Bootstrapped
- 17:50Challenges
- 19:05ETL
- 20:35Problems
- 26:15Numbers
- 29:15Introducing Hadoop
- 31:05Data format
- 32:30Team
- 34:00Conclusion
related talks
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
53mBig Data: from mammoth to elephant
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
45mRuby for the soul of Big Data nerds
48mHyper Speed! When Big Data Blooms
42mWeb Development To Big Data. A Journey
11mBig Data Analytics: Immutability & Idempotence
38mEnabling Big Data and Machine Learning for the Masses...