A practical guide to analysis and interactive visualization of massive datasets
summary
Hands‑on setup of an opinionated Nabari JupyterHub on GCP provisions per‑user servers, then uses Dask (via Dask Gateway) to read ~70 GB (CSV→parquet) flight data, partition into ~641 chunks and scale to ~5–10 workers; shuffles (sorting, set_index) prove expensive so data gets persisted on the cluster. Visuals use HVPlot + Datashader to downsample 125M rows into interactive, widget‑linked plots. Dask task graphs, dashboard and worker memory metrics confirm what works and where time is lost.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 7:56Why Dask is essential
- 13:21Dask data frames basics
- 23:45Dask cluster setup
- 36:24Distributed computing with Dask
- 42:22Minimizing data shuffling
- 49:58Interactive visualizations with HV plot
- 1:00:06Troubleshooting clusters
- 1:07:45Overlaying plots with HV plot
- 1:15:40Geographic plotting with HV plot
- 1:18:52Plotting with DataShader
- 1:23:14Final insights and Q&A on big data visualization
related talks
01h 49mA practical guide to analysis & interactive visualization of massive datasets
03h 07mData of Unusual Size: Interactive Visualization
40mData-centric exploration using intake, dask, hvplot, and more
16mVisualization and Analysis of Large Scale Datasets with Python
53mTips and tricks for robust big data applications
28mProcessing Large Geospatial Datasets with Dask & Xarray
02h 31mTech Tutorial: Data visualization in Python
45mInteractive Analysis of (Large) Financial Data Sets