Big Data. Or Just Too Much Data?
summary
Practical guidance: relational and document engines handle storage but choke on analysis once datasets exceed a few TB—expect a 2–4 terabytes practical limit on commodity hardware because reads start thrashing memory, backups and ALTERs become multi-day operations, and MVCC/ACID plus per-row overhead slow everything. Specific pains include MyRocks compaction costs, B-tree height growth, and skewed cardinality estimates; InnoDB costs about 18 bytes per row overhead while Postgres is higher. For analytics, choose columnar: ClickHouse columnar + SIMD yields far better compression, reads only needed columns, and scales with parallel distributed scans.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:25Overview
- 1:17What is data
- 2:41What is big data
- 4:10How data becomes big
- 7:38How much is too much
- 9:25People suffer when they have bigger datasets
- 10:20Durability of transactional storage
- 10:51What makes it too much
- 16:50Typical asymptotic trend
- 17:53What if I dont want to read it
- 22:05What if I want to do OLTP queries
- 24:00What if I do app queries
- 26:55What about SQL
- 27:48What about PostgreSQL
- 29:58What about MongoDB
- 33:38Conclusion
- 40:15Retention Archiving
- 43:24Disk Memory
related talks
31mBiggish Data With Rails and Postgresql
58mUnderstanding Big Data for Software Engineers
26mBig Data Is Modern Data
26mColumnar Storage: Redefining Data Management for the Modern Era
26mBiggish Data With Rails and Postgresql
36mBig Data, Disruption And The 800 Pound Gorilla In The Corner
49mCan PostgreSQL Compete with Analytics Specialists?
41mAnalyzing Terabytes of Data with Google BigQuery