Testing Parquet Support in ClickHouse Using Combinatorics
summary
Demonstrates that a combinatorial test harness generates real Parquet files with a custom tool parquetify (based on Apache Parquet Java), drives ClickHouse reads via a Python testflows framework, and produces 239,616 combinations for one logical type and about 4,033,472 combinations overall. Shows crashes found (nested arrays with bloom filters), how snapshotting and pairwise reduction tame combinatorial explosion, and that GitHub CI runners run millions of cases in parallel successfully.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:04ClickHouse and Parquet overview
- 3:31Testing Parquet support challenges
- 8:46Inconsistent Parquet structures
- 10:31Custom Parquet files with Parquetify
- 15:06Combinatorics testing approach
- 23:33Combinatorial explosion solutions
- 26:22Test outputs and snapshots
- 28:56Oracles for performance testing
- 31:03State machine exhaustive testing
- 33:13CI/CD pipeline testing
- 34:15Q&A
related talks
47mTesting ClickHouse We Deserve
17mBuilding Real-Time Data Lake with ClickHouse
30mBuilding For Fast
42mTips & Tricks to Keep Your Queries under 100ms with ClickHouse®
34mClickHouse® - What is Behind the Fastest Columnar Database?
04mRoad to Percona Live 2022 - Robert Hodges
26mHandling Large Volumes of Immutable Structured Data With ClickHouse
28mColumnstore Revolution