New Computer Trends and How This Affects Us
summary
Claims that CPU‑level compression plus blocking beats raw I/O by reducing transferred bytes and fitting decompression into CPU latency budgets, with examples: PyTables/HDF5 indexes and out‑of‑core expressions, SSDs on PCIe vs SATA2 bottleneck (~500–550 MB/s), optimal block sizes (memory ~1 KB, SSD ~4 KB, HDD ~1 MB), multi‑core + SIMD use and even 24 cores, and concrete projects built and measured: blos compressor and Bcalls chunked columnar container, group‑by out‑of‑core runs within 2–4× of pandas and blos sometimes exceeds memcpy speeds.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:31Why trends matter now
- 7:19SSD evolution and latency
- 13:43Block access and containers
- 19:34CPU starvation and memory
- 26:08Compression for RAM and storage
- 30:09BColz high performance
- 35:52Blosc 2 future trends
- 38:19Compression's impact
- 38:59Q&A
related talks
25mNew Trends In Storing Large Data Silos With Python
43mImprove the efficiency of your Big Data..
37mHigh Performance Computing: The Modern Approach
39mDensity! The not-so-secret trend driving the future of Data Analytics
28m10 years of Parquet- what's next?
01h 05mPerformance Optimization in Software Development - Being Friendly to Your Hardware
52mI Want to Compress Everything
41mKeynote: How Open Data Science Opens the World of Innovation