Building Distributed Platforms at Extreme Scale on Fast Changing Data
summary
Target builds a platform that tracks inventory across 2,000+ stores and a 4M-item catalog (billions of item‑locations), ingesting ~5,000 supply changes/sec and serving 1,000,000 requests/sec at under 10ms. Initial Cassandra+object cache yields 150–200ms and inconsistency, so the design moves to an in‑memory data grid (Hazelcast/Apache Ignite style), then adds client near‑caches and a custom off‑heap native memory allocator; final system meets latency and consistency goals with serialized formats (ProtoBuf/FlatBuffers), throttling, and off‑heap caching.
AI-written summary · about
chapters (6)
- 0:00Introduction to real-time inventory
- 3:28Platform design goals
- 8:19From disk to memory architecture
- 15:59Custom cache solution
- 18:48Key lessons learned
- 21:19Q&A and closing
related talks
29mEnhancing Flexibility and Productivity with Access Patterns and Storage-Agnostic Abstractions
30mBuilding Stable, Scalable & Agile Distributed Systems at Enterprise Scale
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
47mDesign Decisions Behind Cube Store - a Blazingly Fast Open-Source Rollup Storage Layer
26mBuilding Financial Systems on Eventually Consistent DBs
41mDesign (Failures) Case Study
23mArchitecting for Performance, Scalability and Real-time Data with AWS
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale