Data of Unusual Size: Interactive Visualization

on YouTube ↗03h 07m

" summary

Loads a 70 GB CSV airline set from cloud storage, converts to columnar Parquet, and shows why data types, partitioning and pre-sorting matter for performance. Demonstrations use Dask for distributed computation, hvPlot plus datashader for interactive, rasterized plots of tens of millions of points, and Panel to build servable dashboards. The tutorial shows persisting intermediate results, adaptive scaling to many workers, exercises to reproduce the steps, and live deployment caveats (gateway autoscale hiccups are visible but workarounds and local-cluster fallbacks are provided).

AI-written summary · about

" chapters (15)

1. 0:00Welcome
2. 10:47Setup interactive environment
3. 25:59Airline delays with pandas
4. 35:42Interactive plotting with HoloViews
5. 55:03Nabari platform overview
6. 57:55Dask introduction
7. 1:19:34Dask cluster setup
8. 1:35:20Dask lazy evaluation
9. 1:57:17Storage formats
10. 2:05:48Optimizing data access
11. 2:19:34Visualizing with Datashader
12. 2:30:20Dashboards with Panel
13. 2:44:10Shareable apps with Panel
14. 2:55:23Deployment options
15. 3:03:58Q&A

" related talks

• 01h 25mA practical guide to analysis and interactive visualization of massive datasetsKim Pevey · PyData NYC 2023
• 01h 49mA practical guide to analysis & interactive visualization of massive datasetsEswaramoorthy, Pevey · PyData Global 2023
• 01h 24mData Science at Scale with DaskRichard Pelgrim · PyData London 2022
• 40mData-centric exploration using intake, dask, hvplot, and moreJulia Signell · PyData New York City 2019
• 53mTips and tricks for robust big data applicationsYousry Mohamed · NDC Sydney 2020
• 16mVisualization and Analysis of Large Scale Datasets with PythonChristine Moran · EuroPython 2011
• 30mSeeing the needle AND the haystack: single-datapoint selection for billion-poin...Jean-Luc Stevens · PyCon DE & PyData Berlin 2022
• 01h 23mData Science at Scale with DaskRichard Pelgrim · PyCon DE & PyData Berlin 2022