Optimizing Data-Driven Decisions
summary
Builds an aggregation engine that continuously computes daily stored aggregations so features like “transactions in last 6 months” read at most 180 rows instead of scanning 180 days of raw events, drastically cutting IO. Pipeline uses Airflow + Postgres + queue/stream to populate per-client daily metrics, an aggregator process that pulls messages, runs user-supplied collect and agg functions, and writes metrics. A counter-based metadata check detects lost messages and triggers alerts; API can either error, return partial data, or compute missing days on demand.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker intro
- 2:35Naive aggregation approach
- 5:25Dynamic aggregation needs
- 8:58System architecture overview
- 10:25Metadata management for reliability
- 14:26API for user queries
- 17:00Summary and Q&A
related talks
01h 01mDealing With Data Panel
27mLet Your Data Tell Its Story
25mData Demons: Unleashing Chaos or Driving Performance?
32m7 steps to build-your-own data pipeline - for day 1 of your startup
27mArchitecting Sub-Second Analytics Pipelines
48mHyper Speed! When Big Data Blooms
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
22mPractical approach to building real-time analytics for cybersecurity applications