pandas: Integer NA as a first class citizen

on YouTube ↗31m

" summary

pandas gains a concrete claim: add Integer NA as a first-class dtype via Extension Arrays, implemented as data plus a boolean mask over numpy storage. It preserves integer types through arithmetic, grouping and assignment (most operations work), exposes an opt-in dtype with custom scalars and accessors, and defers copies in the block manager. Known gaps include no ExtensionArrayIndex yet and CSV/serialization roundtrips that still fail.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 1:06Pandas popularity
3. 4:21Pandas limitations
4. 7:13Extension arrays evolution
5. 9:00Integer NA support
6. 12:27Demonstration
7. 18:50Performance implications
8. 21:55Current limitations
9. 22:40Extension arrays use cases
10. 24:28Custom types with pint
11. 27:35Future of extension arrays
12. 28:31Q&A

" related talks

• 28mExtending Pandas with Custom TypesWill Ayd · PyData Los Angeles 2018
• 10mGotchas of PandasPrabhant Singh · PyCon Israel 2017
• 33mExtending Pandas using Apache Arrow and NumbaUwe L. Korn · PyData Berlin 2018
• 33mWhats New In PandasJoris Van den Bossche · PyData Global 2020
• 35mIdiomatic PandasMatt Harrison · Global Artificial Intelligence Virtual Conference 2022
• 27mData Apis: Standardization of N-dimensional arrays and dataframesStephannie Jimenez Gacha · PyCon DE & PyData Berlin 2022
• 01h 17mPandas from the Inside / "Big Pandas"Stephen Simmons · PyData Berlin 2017
• 32mPandas May Be Slow, but Pandas Doesn't Have to BeAndrew Vaccaro · PyColorado 2019