Portable, type-fancy multidimensional arrays
summary
Implements the Zarr spec in portable Scala to store chunked multidimensional arrays (used for single-cell RNA with up to ~2 million cells × ~30,000 genes → ~60 billion entries, ~1 billion nonzeros), provides an NDArray backed by S-lists that encode statically typed rank, supports in-memory and lazy disk-backed chunk layouts, reads/writes Google Cloud Storage, composes with alternative backends, compiles for JVM and JS, and largely works though performance and full slicing / refined bounds checks remain unresolved.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:20Ryan Williams
- 0:56Agenda
- 1:37Overview
- 4:33Gene Expression Matrices
- 11:41Slicing
- 15:33The Dimension Strides
related talks
30mXarray DataStructures in Biology: Examples and Best Practices
25mAn Array-Oriented Language via the Design Recipe
01h 27mMastering Large NDArray Handling with Blosc2 and Caterva2
39mDyND: Enabling complex analytics across the language barrier
20mNumS: Scalable Array Programming for the Cloud
29mEfficient and portable DataFrame storage with Apache Parquet
23mApple Array Allocation
43mImprove the efficiency of your Big Data..