Uncluster Your Data Science Using Vaex
summary
Vaex is an out‑of‑core DataFrame that uses memory mapping, columnar storage and a lazy expression system to run aggregations on a single laptop instead of a cluster. It opens a 110 GB taxi dataset instantly, computes histograms over 1.0+ billion rows in about a second for simple counts and evaluates a heavy spherical distance expression over 1.1 billion points in ~30 seconds, streams Arrow/HDF5/Parquet from S3, and powers interactive Dash/Plotly dashboards and APIs in production.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:50Motivation
- 5:20Vaex
- 6:14Concepts: Memory mapping
- 7:56Concepts: Column based storage
- 9:37Concepts: No memory copies
- 10:50Concepts: Compute & expression system
- 12:23Vaex.io: From academic project to consultancy
- 13:30Demo
- 32:32In production
- 34:23In the wild
- 35:00In production: Dash example
- 36:42Vaex.io: Consultancy
- 37:13Summary
- 37:58Outro
related talks
26mVaex: the perfect DataFrame Library for Python Apps
33mSkinny Pandas Riding On A Rocket
50mFast Analytics On Big Data
40mBeyond Pandas: The Great Python Dataframe Showdown
23mQuery like a Pro: ES|QL meets Apache Arrow and Pandas
43mImprove the efficiency of your Big Data..
30mDask ❤️ Xarray: Geoscience at Massive Scale
42mColumnar Storage + Python: Powering Modern Data Science and Analytics