Visualization and Analysis of Large Scale Datasets with Python
summary
Shows how ParaView plus Python and MPI (example command MPI run) read and render truly large data: 2.2 TB Google n‑grams and 150 GB Wikipedia logs, and even billions of particles interactively. Demonstrates parallel readers, data redistribution, VTK/filters for thresholding and sizing, custom Python readers/writers, and image compositing with IceT. Includes a live example coloring a sphere by process id to confirm data distribution, CSV and binary I/O, and notes about running on clusters or Amazon S3 to avoid massive transfers.
AI-written summary · about
chapters (10)
- 0:00Introduction to workshop
- 0:59Why visualize information
- 1:59Need for parallel processing
- 3:43Paraview and VTK overview
- 5:45Advantages of Python scripting
- 6:52Simple visualization examples
- 8:32Reading and processing data
- 9:38Advanced data analysis
- 10:04Possibilities and conclusion
- 11:43Q&A session
related talks
03h 07mData of Unusual Size: Interactive Visualization
38mPython for Visualization of HPC Applications
01h 25mA practical guide to analysis and interactive visualization of massive datasets
01h 01mParallel Python Analyzing Large Data Sets
41mMassively Parallel Processing with Procedural Python
26mWho Said Wrangling Geospatial Data at Scale was Easy?
01h 49mA practical guide to analysis & interactive visualization of massive datasets
45mInteractive Analysis of (Large) Financial Data Sets