Deconstructing Feather
summary
Claims that Feather is a cross‑language, columnar file format (Wes McKinney Hadley Wickham) that preserves types and metadata and enables zero parsing. Demo writes iris from R reads it in Python then inspects the file: metadata stored at file end (FlatBuffers), per‑column bitmask for nulls, offsets for variable strings and a hex dump. Current state is proof‑of‑concept: read/write works, strings are UTF-8 only, and there is no concatenation or sparse support yet.
AI-written summary · about
chapters (8)
- 0:00Feather file format intro
- 2:31CSV problems
- 3:33Memory hierarchy & caching
- 7:03Feather’s zero-parsing design
- 10:57Live demo: Feather read/write
- 12:43Manual Feather metadata deconstruction
- 17:44Beyond read/write uses
- 20:40Q&A
related talks
37mApache Parquet : the standard, efficient file format for tabular data
08mBackstage interview
10mLet me at my data! Pulling pictures out of an undocumented file format
31mState of Parquet 2025: Structure, Optimizations, and Recent Innovations
36mEmbrace the chaos: Composable data systems with fewer asterisks
58mComposable Data Systems, the new frontier
42mMaking Sense Out of Flight Test Data with Python
24mPolyglot data with python: Introducing Pandas and Apache Arrow