Massively Parallel Processing with Procedural Python
summary
Claims that Python runs inside the database to avoid moving terabytes of data to memory, using PL/Python to call numpy scikit-learn and NLTK on nodes and MADlib for cluster-wide models. Shows a Twitter pipeline ingesting 55 million tweets per day, an image-processing pipeline storing pixels as rows, and a traffic-incident predictor backed by Greenplum / Pivotal HAWQ; demos mostly work but live examples require cluster libraries and network access to reproduce.
AI-written summary · about
chapters (9)
- 0:00Massively parallel Python processing
- 4:27PL Python core benefits
- 11:57Returning rows and columns in PL Python
- 15:49Parallelizing PL Python with groups
- 19:01MADlib for scalable ML
- 21:28PL Python image processing use case
- 24:20Social media analysis with PL Python
- 28:58Traffic incident prediction demo
- 32:36Q&A
related talks
40mScaling Python - Bank Edition
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
37mAccelerating Python Analytics by In-Database Processing
35mData Processing Pipelines for Small Big Data
26mPython + MPP Database = Large Scale AI/ML Projects in Production Faster
44mUsing Python with Hadoop to create production ready big data applications
01h 01mParallel Python Analyzing Large Data Sets
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas