Strengthening Apache Pinot's Query Processing Engine with Adaptive Server...
summary
Apache Pinot at LinkedIn adds Adaptive server selection and Runtime query killing to handle 1,500+ brokers, 8,000+ servers, 4,500 tables and ~300K QPS. Adaptive selection uses per-broker moving-average stats and a latency×(estimatedQsize^alpha) score to avoid thundering‑herd routing, cutting slowness alerts and preventing latency degradation for >90% of server underperformance events with sub‑millisecond stat overhead. Runtime killing uses thread‑local query contexts and a lockless sampling aggregator to preempt OOMs, catching >85% crashes with ~3% CPU cost.
AI-written summary · about
chapters (13)
- 0:00Pinot and LinkedIn use
- 3:58Pinot’s resiliency journey
- 7:55Static round-robin routing issues
- 10:28Adaptive server selection design
- 16:24Server selection policy
- 19:21Adaptive routing performance
- 22:10Production results and benefits
- 22:54Need for automated query killing
- 27:56Automatic query killing design
- 31:33Query execution context tracking
- 34:05Query accounting and killing logic
- 36:19Operational benefits and benchmarks
- 38:05Future work and closing
related talks
54mAn Introduction to Apache Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
49mIntroduction to Real-Time Analytics with Apache Pinot
37mCan Apache Pinot replace your OLTP database?
34mNext generation OLAP stack using Apache Pinot
33mPinot, Why Are You So Fast?
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
47mReal-time analytics for your users with Apache Pinot