Dealing With Data Panel
summary
Companies report concrete approaches: ASPSFlyer-ingest pipeline handles 70 billion events/day into Kafka then Parquet on S3 (35–40 TB/day compressed) and runs Spark aggregations; Metabase targets everything from 100-row Heroku DBs up to 100 billion rows, explores notebook-driven ETL and in‑tool prediction UX; teams use cardinality limits, precomputed slices vs on‑the‑fly aggregation, and bi‑temporal tooling like Crux bi‑temporal views. Common asks are declarative ETL, streaming event logs, materialized views, and safer co‑located computation.
AI-written summary · about
chapters (10)
- 0:00Panel introduction
- 1:45Metabase data challenges
- 7:03AppsFlyer scale and trade-offs
- 13:05Crux and bi-temporal data
- 23:21Metabase user empowerment
- 30:02Ideal data interaction
- 38:46Future of streaming data
- 44:22Future of ETL
- 51:47NoSQL and computation
- 1:00:50Q&A and closing
related talks
24mGive meaning to 100 billion analytics events a day, analytics at Teads
38mHow to handle 100k events per second
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
17mDemocratising data analytics
48mHyper Speed! When Big Data Blooms
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
17mOptimizing Data-Driven Decisions