Fast Access To Your Complex Data #bbuzz
summary
Benchmarks run on Apache Spark compare Avro JSON ORC and Parquet across real datasets: 22 million taxi rows, 25 million synthetic sales rows, and 10 million GitHub rows with 704 columns. JSON is consistently the slowest and largest; Parquet often gives the smallest files with zlib and the fastest full-table reads; ORC gains dramatic speed with the native/vectorized reader and predicate pushdown, reading as little as 10,000-row predicate granularity versus Parquet reading whole stripes unless you turn on orc native=true and enable the proper filters.
AI-written summary · about
chapters (11)
- 0:00Introduction and goals
- 2:58File format characteristics
- 6:12Benchmark datasets
- 9:25Software and configuration
- 12:30Storage and read benchmarks
- 16:48Performance analysis
- 20:43Column projection
- 22:56Predicate pushdown
- 25:24Benchmark results and insights
- 29:36Best practices and conclusions
- 33:39Q&A session
related talks
20mGive me the Data
29mEfficient and portable DataFrame storage with Apache Parquet
22mAccelerating Analytics with Apache Arrow
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
46mEfficient Data formats for Analytics with Parquet and Arrow
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
42mColumnar Storage: Redefining Data Management for the Modern Era