Particle physics, 10,000 times faster
summary
Claim: nested-event queries compile into columnar array operations and run orders of magnitude faster. Implementation uses SparkSQL Impala Kudu ROOT Parquet Arrow and a Python-to-Numba code transform that emits offset arrays instead of objects. Result: Femtocode and the array-transform pipeline deliver 10,000 times faster simple queries versus the legacy C++ framework on the measured workloads. Key technique is Python+Numba array transform and offset-based shredding; benchmarks and storage-format details are provided.
AI-written summary · about
chapters (7)
- 0:00Intro
- 4:48Data is stored in files
- 5:55Data structure
- 13:03Query language
- 19:44Data as columns
- 24:54Code transformation
- 27:26Data in columns
related talks
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
15m100x faster data app experiences delivered cost-effectively
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
26mColumnar Storage: Redefining Data Management for the Modern Era
42mColumnar Storage: Redefining Data Management for the Modern Era
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mInternals of Speeding up PySpark with Arrow