How do you store your data?
summary
Describes a production path: started with MySQL proxying translated pages (works for 2–3 small sites, collapses around 10 req/s), moves to in-memory caches and a second nano-latency cache, experiments with using Lucene indexes as primary store, then switches heavy HTML storage to MongoDB (3 shards × 3 replicas) which later suffers write contention and disk I/O; solution is 32 shards on 64 nodes, handling ~2 GB/min writes, 13 TB total, ~150 GB/day through that module. Argues for polyglot persistence, shows CAP tradeoffs and practical limits of each system.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:17Relational databases limits
- 3:46NoSQL databases emergence
- 5:13NoSQL types
- 8:54Traditional solutions problems
- 11:04Data tech selection
- 13:13Caching and performance
- 17:37HTML content storage
- 21:11Storage lessons
- 22:56Polyglot persistence
- 24:40CAP theorem myths
- 32:35Complex data models
- 33:37Graph data models
- 36:51Graph DB advantages
- 38:17ArangoDB features
- 48:34NoSQL risks
- 50:19File handling issues
- 57:02Query tools and scaling
- 1:03:59Q&A
related talks
25mData in the Cloud
52mData storage for modern applications - Mike Lehan
54mScaling up and accelerating Drupal 8 with NoSQL
43mCaching and Beyond - Smarter Data Processing with Java
34mData Storage: NoSQL Toasters and a Cloud of Kitchen Sinks
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
52mTaking Storage for a Ride
46mEdge Performance with In memory NoSQL