Columnar Storage: Redefining Data Management for the Modern Era
summary
Columnar storage claim: concrete speed and scale wins using Apache Arrow (in-memory), Apache Parquet (file), and libraries like Pandas or DuckDB. Examples: ClickHouse reads Parquet (not as its native store), InfluxDB uses Arrow+Parquet, Cloudflare ingests ~8M rows/sec, Disney+ logs ~2 billion rows/sec, observability use sees ~15M queries/day over petabytes. Demonstrations show Arrow Flight removes serialization cost, Parquet enables file-level data skipping, and real deployments cut query latency and storage cost.
AI-written summary · about
chapters (11)
- 0:00Columnar storage introduction
- 1:30Columnar database features
- 3:58Row vs column storage
- 7:51Use cases for row vs columnar
- 10:58Real-world columnar apps
- 14:49Quiz on columnar workloads
- 21:34Apache Arrow & Parquet formats
- 27:59Arrow benefits for analytics
- 33:44Arrow Flight & SQL integration
- 38:42Analytics tools with columnar storage
- 41:29Conclusion
related talks
26mColumnar Storage: Redefining Data Management for the Modern Era
46mEfficient Data formats for Analytics with Parquet and Arrow
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
27mApache Arrow - The columnar format! What's that?
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
33mArrow in Flight - New Developments in Data Connectivity
35mPractical Applications of Apache Arrow
42mColumnar Storage + Python: Powering Modern Data Science and Analytics