Pure Ruby Apache Arrow reader/writer
summary
A pure-Ruby reader/writer for the Apache Arrow file/stream format (published as red_arrow_format) reads and writes Arrow with zero-copy where possible by using Ruby 3.1’s IOBuffer and IOBuffer.map (mmap) and includes a side-project: a FlatBuffers in Ruby implementation. Benchmarks show the pure-Ruby reader matches a C debug build, but a tuned C/C++ release is several times faster; writing is close — roughly twice slower than optimized C yet noticeably faster than the C debug runs.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 3:49Pure Ruby Arrow library
- 7:58Why Apache Arrow is fast
- 11:49Zero-copy in Ruby techniques
- 18:11Memory mapping for speed
- 19:37Writing performance
- 21:53Adopting Arrow in Ruby
- 26:55Call to action and resources
related talks
27mApache Arrow - The columnar format! What's that?
09mBackstage interview
39mDesigning for high-performance execution with Arrow and Polars
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
46mInternals of Speeding up PySpark with Arrow
28mHigh speed data from the Lakehouse to DataFrames with Apache Arrow
24mPolyglot data with python: Introducing Pandas and Apache Arrow
31mThe Arrow revolution in pandas and Dask