Optimizing Latency-Sensitive Queries for Presto at Facebook
summary
Facebook embeds Alluxio local on-SSD cache into Presto workers and measures results on a 600-node shadow cluster with 460 GB cache per node and 17,000 real queries. Using 1MB segment blocks, LRU eviction and async background writes, p95 latencies drop roughly 50% and IO scan volume falls from 582 TB to 250 TB (~57% scan savings). The cache is segment-based, off-heap, supports pluggable policies and RocksDB storage, handles disk failures, and runs in production.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:26Presto @ Facebook Scale
- 1:26How Presto Works
- 6:21Affinity Scheduling
- 11:00File Merge Caching
- 12:01Learnings & Alluxio Collaboration
- 14:05Benchmark Configuration
- 14:38Benchmark Results
- 16:29Alluxio Overview Translate access to optimal storage APIs over a slow network
- 17:59Options with Alluxio
- 19:11Presto & Alluxio Local Cache Architecture
- 20:07Implementation & Optimization
- 22:39Ongoing Development (WIP) Support Schema/Table/Partition level Cache Quota
- 24:02Alluxio File System Enhancements
related talks
20mData Caching for Enterprise-Grade Petabyte-Scale OLAP
29mA Case Study in API Cost of Running Analytics in the Cloud at Scale with an...
46mPresto - Today and Beyond - The Open Source SQL Engine for Querying all Data Lakes
22mPresto for Your Fast, Open SQL Lakehouse
49mPresto in the Cloud
51mPresto Open Source SQL Engine for Massive Scale Data Lakehouses
22mIntroduction to Presto: The SQL Engine for Data Platform Teams
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...