Segmenting 600 Million Users

on YouTube ↗43m

" summary

Describes a system that performs real-time segmentation of 600 million users daily using streaming processing. Covers delivering segment data to partners, DSPs and Google with minimal latency, maintaining counts of unique users per segment in streaming, tracking effects of segmentation parameter changes, and computing domain-to-segment affinity for audience analytics.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 3:36Old system problems
3. 9:11Real-time segmentation shift
4. 10:15HBase real-time optimization
5. 11:40Kafka integration
6. 13:03Segmentation scaling
7. 18:08HBase load issues
8. 22:25Timestream optimization
9. 25:41Bots and performance
10. 27:08Monitoring and resilience
11. 31:47Summary and next steps
12. 32:28Q&A

" related talks

• 23mStream Algorithms for Big Data Processing TasksViktor Evstratov · HighLoad++ 2015
• 24mReal-Time Streaming Segmentation on Azuresand naate · Aerospike Summit 2023
• 22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million VisitorsFelix Gessert · code.talks 2023
• 20mStreaming Processing Options in Google CloudSergei Sokolenko · Flink Forward Europe 2019
• 04mHighlights - Data and BusinessW.Vogels, J.Zavalishina, C.Moody, T.Haile, N.Carlson · DLD 2015
• 56mSpark-driven Audience countingBoris Trofimov · JavaDay Kharkiv 2015
• 24mEfficient Similar-Object Search Algorithms for Terabytes of DataEvgeniy Zhurin · HighLoad++ 2015
• 19mLog Events @ Twitter: Challenges of Handling Billions of Events per MinuteLohit Vijayarenu · Systems @Scale Fall 2021