Using Hadoop at Badoo
summary
Badoo describes using Hadoop to store and process the entire clickstream from the web site and mobile app servers—about 1 billion events per day—and an additional ~1 billion client- and server-side user action events. The talk covers how they organize and run this Hadoop-based pipeline and what kinds of analyses and uses are feasible or impractical with Hadoop.
AI-written summary · about
chapters (25)
- 0:00Hadoop at Badoo intro
- 3:39Why Badoo needed Hadoop
- 7:44Starting with Hadoop setup
- 11:17Hadoop architecture basics
- 14:27Storing user activity in Hadoop
- 21:14Aggregating activity data
- 22:19Monitoring Hadoop cluster
- 25:55Hadoop challenges and limits
- 26:38Future plans Spark updates
- 27:43Cluster config and data volumes
- 30:16Q&A Text vs binary formats
- 32:24Q&A Buffering logs Cassandra
- 34:50Q&A Cluster growth scaling
- 35:56Q&A MapReduce usage alternatives
- 37:00Q&A Hadoop updates maintenance
- 38:06Q&A Activity event structure
- 39:54Q&A Backup strategies testing
- 40:41Q&A Ganglia for monitoring
- 41:47Q&A Disk usage replication
- 44:01Q&A Failover solutions YARN
- 45:46Q&A Shark Spark plans
- 46:30Q&A Partitioning by date
- 47:53Q&A Moving data to databases
- 49:20Q&A Cluster downtime config
- 50:01Q&A Data retention limits
