Democratising Attention Data at theguardian.com
summary
Built an internal real‑time analytics system called ophan that evolves from a 24‑hour hack into production: JavaScript pixel → SNS/SQS → a Play/Scala dashboard first using an in‑memory three-minute in-memory window then backed by an ElasticSearch cluster, deployed with CloudFormation and Auto Scaling. Two engineers part‑time process about 15–25 million events/day, expose simple page‑views and referral bars to 400+ staff, and use the data to spot spikes, remove bad ads and change front‑page promotion in minutes.
AI-written summary · about
chapters (8)
- 0:00Democratizing attention data
- 3:54Live demo of attention tool
- 12:47Tool origins in hack day
- 21:47Prototype to production
- 30:33Infrastructure and ownership
- 37:23Driving engagement with data
- 47:21Conclusion and lessons
- 48:26Q&A on open source
related talks
11mData science: Next-gen performance analytics
12mUnlocking the Secrets of Client Data
01h 07mBig_Data_Meets_Digital_Advertising
38mBuilding an Analytics System for a High‑Load Media Platform
30mDebugging and Optimizing User Experience with Deep Visual Analytics
27mHow to be data-driven when you aren’t Netflix (or even if you are)
38mDeveloping an Analytics System for a High-Load Media Service
35mVisualising Public Service: Seeing Inside Birmingham City