Pinterest’s Exploration of Distributed SQL
summary
Pinterest runs HBase across roughly 50 clusters storing over 10+ PB and serving ~100M+ QPS with many systems on top (graph DB “Zen”, column store UMS, RocksDB K/V, Muse search, Druid, Goku). Middleware adds transactions and secondary indexes but causes complexity, inconsistencies and 24‑hour index rechecks. After evaluating ~15 options, open‑source NewSQL candidates stood out and the team runs dark‑traffic migration tests (schema compatibility, online migration, validation) to measure performance, hotspots, operational costs and reliability before any cutover.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:11Agenda
- 0:38The Evolution of Pinterest Online Storage
- 2:41Overview about Hbase and Pinterest
- 4:27Index Data Store
- 5:02Storage Systems
- 5:35The Architecture of the Edge Base Ecosystem
- 6:21Hbase Replication Proxy
- 7:49High Complexity
- 8:25Challenge Is Data Inconsistencies
- 9:13Tail Latency
- 10:28Major Pain Points of Zen
- 16:24Performance Benchmarking
- 19:38System Maturity
- 20:45Operational Load
- 21:48Early Learnings
related talks
25mHundreds of Terabytes of Data
54mProxySQL at Scale
23mZen: Pinterest's Graph Storage Service
19mBuilding KVStore for ML Workloads
54mAn Introduction to Apache Pinot
37mCan Apache Pinot replace your OLTP database?
29mModern Database Architecture: From SQL to NoSQL & Distributed SQL
26mNewSQL: concepts, realities and when you should embrace it