Contemporary Approaches to Data at Scale
summary
Explains that storage performance hinges on sequential vs random I/O and compares three concrete engines: in-memory indexes (fast until indexes exceed RAM), log-structured merge-tree collections (Cassandra/HBase style: append-only SSTables plus compaction), and columnar files (Parquet/column stores for compressed analytics). Covers Kafka as a sequential log, contrasts consistent-hash sharding with broadcast analytics, and advocates a stream-first architecture combining append-only logs, stream processors, and replicas plus sharding for scalable, reproducible pipelines.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:55Locality
- 6:10Indexing
- 7:33Random workload
- 9:32What does that gain
- 10:11Optimize read
- 11:36Log structured merge tree
- 12:32Brute force approach
- 14:12Merge join
- 14:58Compression
- 16:04Kafka
- 17:03ReactMongo
- 17:44Big Data Tools
- 18:15Storage Engines
- 18:38Parallelism
- 19:08Hashing
- 19:50Broadcast
- 21:21Replication
- 23:29Single Writer Principle
- 25:17CQRS
- 26:28Druid
- 27:34Database Inside Out
- 32:24Stream Data Platform
- 34:26Source Systems
- 36:34Routing State
- 38:18Single Store
- 40:06DeTomic
related talks
58mUnderstanding Big Data for Software Engineers
27mNavigating Unstructured Data - Availability vs Analytics in NoSQL
35mQuestioning Database Claims: Design Patterns of Storage Engine
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
42mColumnar Storage: Redefining Data Management for the Modern Era
26mColumnar Storage: Redefining Data Management for the Modern Era
47mWhat's Ahead for Big Data?
34mData Storage: NoSQL Toasters and a Cloud of Kitchen Sinks