A 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
summary
Claims concrete performance: using Arrow Flight, DataFusion, Apache Arrow and Parquet 87% smaller yields real speedups — a Spark round trip drops serialization time by ~84% with Arrow, Parquet compresses 1 TB CSV to ~130 GB and queries 34× faster, and a demo million-row Pokémon dataset returns sub-millisecond queries. Code writes and reads Parquet, converts Arrow ↔ pandas/polars, and a working pipeline is built: Flight SQL → Grafana for visualization and an Arrow-based task engine runs anomaly detection on robot vibration data and surfaces faults.
AI-written summary · about
chapters (16)
- 0:00Time series analytics intro
- 1:54Evolution of time series data
- 5:09Traditional challenges
- 8:38FDAP stack
- 10:25Arrow Flight and SQL
- 14:02DataFusion query engine
- 16:11Arrow in-memory format
- 18:17Parquet storage
- 21:06Arrow vs Spark proof
- 23:38Arrow tables
- 25:42Arrow Flight demo
- 29:11Pandas vs Polars
- 32:06Simple FDAP database
- 38:27Anomaly detection use case
- 42:42Projects and resources
- 45:00Q&A
related talks
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
28mHigh speed data from the Lakehouse to DataFrames with Apache Arrow
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
31mThe Arrow revolution in pandas and Dask
42mColumnar Storage: Redefining Data Management for the Modern Era
29mFaster PySpark with Apache Arrow