Multiplying really really big matrices
summary
Shows a concrete pipeline: convert sequences to k‑mer count vectors, store huge column‑wise database as numpy.memmap files, assemble each column slice into a CSR sparse slice and run slice‑wise matrix multiplication by column slices against dense queries. Code uses numpy, scipi sparse APIs, Cython bits and Biopython for IO. Implementation runs on a laptop and is usable for prototyping, but lacks thorough benchmarks and a clean SciPy‑style interface.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 5:10D2 statistic and matrix multiplication
- 7:41Large sparse matrices in memory
- 9:07NumPy mmap for out-of-memory data
- 10:34Matrix slicing and mmap strategy
- 12:46Building database with mmap slices
- 16:48Querying with matrix multiplication
- 19:24Performance and alternatives
- 20:32Q&A
related talks
27mSpeed up Python with hardware accelerators
34mNumPy: vectorize your brain
29mLarge-Scale Matrix Operations
27mAPL-Inspired Techniques for Advanced NumPy
20mNumS: Scalable Array Programming for the Cloud
42mPython’s role in accelerating biomedical science
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
01h 01mParallel Python Analyzing Large Data Sets