When Metrics Lie
summary
Shows concrete failures at Pinterest where logging bugs flip a measured 5% lift into no effect or a loss and a deployed dashboard that silently omits events. Demonstrates practical fixes: end-to-end logging tests, a backfill and replay tool that verifies experiment telemetry, automated event lineage and diffs between raw and aggregated data, and lightweight shadow experiments that validate metrics before rollout so fixes actually stop false conclusions.
AI-written summary · about
chapters (5)
- 0:00Misleading metrics intro
- 7:54Web analytics pitfalls
- 16:03Metric deception examples
- 23:08Avoiding data deception
- 24:44Conclusion
related talks
23mYou Get What You Measure—Why Metrics Are Important
01h 01mFinding Metrics that matter and using them safely
37mYou Can't Optimize What You Can't Measure
10mOf Mice and Metrics
36mMetrics - What's your code actually doing?
43mObservability for Emerging Infra: What Got You Here Won't Get You There
28mProduct metrics for developers
01h 01mKeynote - Starting a metrics program