Visualization and Analysis of Large Scale Datasets with Python

on YouTube ↗16m

" summary

Shows how to script ParaView with Python to run MPI servers, use parallel readers and a data distribution filter, and render with IceT compositing so each process renders local data and images composite correctly. Demonstrates coloring a sphere by process to verify distribution, reads 2.2 TB ngrams and 150 GB Wikipedia examples, handles billions of particles and thousand‑core runs, and builds custom readers and filters in Python or C++ while keeping ParaView under a BSD license.

AI-written summary · about

" chapters (14)

1. 0:00Parallel visualization intro
2. 1:12Why visualize data
3. 2:16Need for parallel processing
4. 3:42Paraview architecture
5. 4:24Python with Paraview
6. 6:53Tutorial and demos
7. 8:16Debug data distribution
8. 8:37Reading CSV files
9. 9:20Random data generation
10. 9:41Stellar cluster data
11. 10:24Large dataset handling
12. 11:28Q&A data sources
13. 13:13Q&A scientific visualization
14. 14:46Q&A platform independence

" related talks

• 41mMassively Parallel Processing with Procedural PythonRonert Obst · PyData Berlin 2014
• 10mMPI Meets Python: How to Use MPI in the Python UniverseKawthar Shafie Khorassani, Arpan Jain · PyOhio 2020
• 01h 01mParallel Python Analyzing Large Data SetsMatthew Rocklin, Aron Ahmadia · PyData DC 2016
• 01h 31mModern scientific computing and big data analytics in PythonEdward Schofield · PyCon Australia 2013
• 01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in PythonEhsan Totoni · Vaccination Database Talks (Second Dose) 2021
• 45mInteractive Analysis of (Large) Financial Data SetsYves J. Hilpisch · PyData Berlin 2014
• 01h 04mVisualising Data through PandasVincent D Warmerdam · PyData Berlin 2014
• 44mUsing Python with Hadoop to create production ready big data applicationsUlrich Zink · PyData London 2016