Fully Automated HW SKU Selection System to Optimize Apache Pinot’s Cost-to...
summary
Claims a 35% reduction in cost-to-serve by migrating 20% of clusters and a 57% cut to future hardware forecasts, after aiming for ~20% savings; builds an orchestrator, migrator, cost estimator (linear programming over SKU capacities), smart provisioning and benchmarking service; collects subsecond resource profiles with PCP collection at 15–50ms, compresses distributions with T-digest percentile compression, streams data via Kafka into Pinot, models CPU/memory/IO headroom and SSD IOPS/throughput, and runs dual-SKU rollouts that validate latency before disabling old replicas.
AI-written summary · about
chapters (11)
- 0:00Apache Pinot introduction
- 3:12Single SKU challenges
- 7:09SKU system goals
- 8:54Cost optimization approach
- 12:37Memory and storage metrics
- 16:53Metrics storage with T-digest
- 21:09Linear programming for SKU
- 23:22Zero-downtime SKU migration
- 26:13Multi-SKU smart provisioning
- 29:21Results and cost savings
- 30:04Acknowledgments and Q&A
related talks
54mAn Introduction to Apache Pinot
40mStrengthening Apache Pinot's Query Processing Engine with Adaptive Server...
37mCan Apache Pinot replace your OLTP database?
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
34mNext generation OLAP stack using Apache Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
33mPinot, Why Are You So Fast?
49mIntroduction to Real-Time Analytics with Apache Pinot