Building KVStore for ML Workloads
summary
Pinterest runs KVStore, a C++ distributed key-value store built on RosDB with Helix shard management, ingesting bulk files from S3 and streaming from Kafka or Flink. It serves low-latency ML features across ~28,000 EC2 VMs, processes 117M requests/sec, stores >10 PB on SSDs and ingests ~5 PB/day. Read-only version swaps S3-loaded versions; mutable mode async-replicates writes to leaders and followers. Network pacing, per-tenant quotas, tunable replication and SSD compression drive reliability and cost efficiency.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:09What are ML features
- 2:16What is KVStore
- 4:04Overview of KVStore
- 5:26KVStore Internals
- 7:26Data Model API
- 8:34Network Throttling
- 10:55Noise Neighbor Problems
- 12:34Cost Efficiency
- 16:13Recap
- 16:54Future Trends
related talks
18mSplitKV: Splitting IO Paths for Different Sized Key-Value Items with Advanced
12mSpanDB: A Fast, Cost-Effective LSM-tree Based KV Store on Hybrid Storage
24mUsing a Distributed Key-Value Store
24mPinterest’s Exploration of Distributed SQL
30mStateful Superpowers: Explore High Performa...
21mMatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores
14mLoLKV: The Logless, Linearizable, RDMA-based Key-Value Storage System
44mFeature store