Avoiding Big Data Antipatterns
summary
Covers concrete big-data mistakes and fixes: choosing Hadoop for a 20 GB set wastes time and ops, partitioning and columnar Parquet files plus bucketed sort-merge joins cut I/O; Kafka originates as LinkedIn’s data-integration layer and benchmarks show 2M TPS with ~2 ms latency; Cassandra’s soft deletes leave tombstones by default for 10 days and wreck scans unless schema and retention are redesigned; use HyperLogLog-like sketches to deduplicate — ~1.5 kilobits per billion distinct — and enable production-grade security.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:16Agenda
- 4:36A Single Tool for the Job
- 8:06Data Integration
- 9:52Hallmark of Kafka
- 12:25Hadoop Was Designed To Efficiently Scan over Large Amounts of Data
- 12:53Data Locality
- 13:35Partition the Data
- 14:12Projections
- 14:46Harsh Join
- 16:21Using a Corner Data Format
- 17:15Tombstones
- 26:59Summary
related talks
29mBig Data Analysis
01h 00mBig Data, Bad Analogies
40mTop 10 data engineering mistakes #bbuzz
56mResistance is Futile: What You Need to Know about Big Data
15mOptimizing Query Performance in Big Data Systems
49mBig Data Architectures and Ten Typical Pitfalls Along the Way
47mSolving Real & Big (Data) Problems Using Hadoop
02h 18mDistributed Systems: In the Daily Life of the Data Engineer