Video Access-Log Processing with Apache Flink

on YouTube ↗22m

" summary

Mux processes CDN access logs to meter per-account, per-asset video usage and power no-commit utility billing by enriching logs, windowing them by event time, and writing hourly usage to Postgres. The pipeline uses Apache Flink for exactly-once enrichment and event-time windows with three hours allowed lateness, Kafka for raw and enriched topics, and Redis + CockroachDB for asset lookups. The system scales by changing Flink parallelism and survived large spikes without double-billing.

AI-written summary · about

" chapters (13)

1. 0:00Intro
2. 0:33Agenda
3. 2:37Web Video Primer: Media Request Sequence
4. 4:14Web Video Primer: System Interactions
5. 4:45What is Utility Billing?
6. 5:15Challenges with Utility Billing:Log Enrichment
7. 5:34Utility Billing Challenges: Thundering Herd Problem
8. 6:11Utility Billing Challenges: Late/Unordered Data
9. 7:03Serve the Video
10. 8:41Step 2: Collect CDN Access Logs
11. 10:48Step 3: Enrich Access Logs
12. 15:46Utility Billing Usage Calculations
13. 17:16Summary

" related talks

• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 19mStreaming Event-Time Partitioning With Apache Flink and Apache IcebergJulia Bennett · Flink Forward Europe 2019
• 27mIntroduction to Flink in 30 minutesDavid Anderson, Johannes Moser · Flink Forward Global 2021
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 40mRobust Stream Processing with Apache FlinkJamie Grier · Flink Forward Berlin 2016
• 49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing ApplicationsStephan Ewen · Scale By the Bay 2017