Mastering Large NDArray Handling with Blosc2 and Caterva2

on YouTube ↗01h 27m

" summary

Blosc2 is a compression format and library for large NDArrays that shows concrete results: a 10,000×10,000 array (100 million elements) compresses to a few megabytes, often ~100x compression and examples where a saved array is ~4 MB for 100M elements. The runtime supports lazy expressions saved to disk and reductions, user‑defined functions accelerated with Numba, chunk/block tuning, and Caterva2 for remote sharing and serving (demoed with a 3D tomography set of 10 images).

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 1:30Blosc2 and Caterva2 overview
3. 7:00NDArray tutorial setup
4. 10:04Creating NDArrays in Blosc2
5. 27:16Blosc2 vs Iron Array
6. 31:29Lazy arrays in Blosc2
7. 51:51Querying structured arrays
8. 1:01:54UDFs with Blosc2 and Numba
9. 1:09:54Caterva2 for remote access
10. 1:20:47Q&A

" related talks

• 43mImprove the efficiency of your Big Data..Francesc Alted, Christian Steiner · PyData New York City 2019
• 22mBlosc2: Fast And Flexible Handling Of N-Dimensional and Sparse DatasetsFrancesc Alted · PyData Global 2023
• 25mNew Trends In Storing Large Data Silos With PythonFrancesc Alted · EuroPython 2015
• 33mPortable, type-fancy multidimensional arraysRyan Williams · Typelevel Summit Philadelphia 2019
• 01h 20mAdvanced Data StorageThomas Pfaff · PyData Berlin 2015
• 20mNumS: Scalable Array Programming for the CloudMelih Elibol · Ray Summit 2021
• 26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...Itamar Turner Trauring · PyCon US 2020
• 30mHow to grow your own distributed array databaseAlice Liang · Clojure/West 2014