Big Data Analytics With AWS Athena
summary
A custom ETL of gzipped CDN logs plus MongoDB session and event collections gets replaced by Athena querying S3 tables, collapsing the join-and-index nightmare and turning a typical join that took 15 minutes → 10 seconds into an interactive query. Collection uses three beacon types consolidated into a magic beacon, then views convert nested JSON timestamps and resolve user agents, sessions are deduplicated and aggregated for KPI correlation with performance, and QuickSight caused spice timeouts and query rewrites that broke reliable dashboards.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:30I Am Wolle
- 1:09I Like Real-Time Stuff
- 2:54Let's Talk About Batch Analytics
- 3:493 Things Make Your Website Slow
- 4:24We Make Websites Fast
- 5:25Accelerating Personalized HTML
- 6:14Split Testing for Web Performance
- 7:25Goal: Performance & Business Insights
- 8:13How to Monitor Performance?
- 9:42When to Send Data Beacons ?
- 10:45Types of Data Beacons
- 11:51Tracking Data in MongoDB
- 14:01Problems I: CDN Data Import
- 15:32Problems li: Aggregation Pipeline
- 16:31Problems II: Reporting
- 18:02Problems IV: Joins in MongoDB
- 20:11What's an Athena ?
- 21:10Upgrading Our ETL Pipeline
- 21:58Processing Stages & Latency
- 23:48Example: Timestamps
- 25:33Join Beacons
- 26:05Resolve User Agents
- 27:17Session Metrics
- 29:05Materialized Views
- 29:34Reporting: The Right Tool for the Job
- 32:05Time Works Differently in AWS Dashboards
- 33:07Click Once, Pay Twice
- 36:54Real-Time Analytics & Reporting
related talks
47mBuilding Near Real-time Analytics Solution on AWS
28mServerless Data Analysis and Reporting using SQL
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos
27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
24mTurning Data into Insight
39mBuild Your Data Lake on AWS — The Flo Experience
43mAnalytics capabilities architectural patterns: A journey for data-driven insights
37mBuild a Modern Data Platform on AWS