Kudu
summary
Kudu is a storage engine positioned between HDFS and HBase that integrates with Impala and targets fast analytical scans plus low-latency updates. A 50 GB CSV workload shrinks to 50 GB → 27 GB with encoding (claims of metadata dropping to a few MB), inserts can slow from ~23M/sec to ~2.8M/sec, while some query latencies fall from 16s → 1s. Production caveats include unreadable GDBC errors, corruption cases (“bad magic”), no foreign keys or autoincrement, no vertical disk expansion, and tight Hadoop/Impala dependencies.
AI-written summary · about
chapters (21)
- 0:00Introduction and background
- 1:11Kudu in storage and Hadoop stack
- 3:39Scaling SQL databases pain points
- 5:04Kudu performance and use cases
- 6:50Excelme architecture with Kudu
- 8:00Replication and CP theorem
- 9:07Horizontal vs vertical scaling limits
- 10:14Multi-level partitioning explained
- 12:40Data compression and encoding
- 14:54Performance test results
- 16:43MySQL to Kudu migration benefits
- 17:24When to use and avoid Kudu
- 18:51Pros and cons overview
- 20:18Missing features and limitations
- 22:49Error handling and corruption issues
- 25:23Distributed system quirks
- 27:13Production readiness concerns
- 28:16Future outlook and recommendations
- 29:26Comparison with Google Cloud Spanner
- 31:39GDPR and data center placement
- 32:27Q&A
related talks
40mApache Kudu (incubating): Fast Analytics on Fast Data #bbuzz
45mUsing Apache Kudu for Analytical Workloads
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
31mBuilding large scale, transactional data lakes with Apache Hudi
45mAnalytics for not so big data with DuckDB
31mData wrangling like a boss with Duckdb
41mAnalyzing Terabytes of Data with Google BigQuery
50mApache Hudi : The Data Lake Platform