Python MapReduce Programming with Pydoop
summary
A Python API called Pydoop exposes full MapReduce components (mappers, reducers, record readers, partitioners) by wrapping the Hadoop Pipes and HDFS C APIs with C++/Python extensions, so Python programs talk to Hadoop over persistent sockets. Benchmarks show performance close to Java/Pipes and much faster than streaming. It runs in production for genomics: six Illumina sequencers produce about 4 TB/week and a genotype-calling workflow for 7,000 individuals drops from two weeks to under one day.
AI-written summary · about
chapters (14)
- 0:00MapReduce and parallel programming
- 3:00Why MapReduce for big data
- 10:17MapReduce in distributed systems
- 18:57Hadoop and HDFS explained
- 25:42Single-node Hadoop setup
- 31:40WordCount example in Hadoop
- 34:36Challenges in Python MapReduce
- 39:15Pydoop introduction and benefits
- 43:36Pydoop API design
- 47:38Custom components and counters with Pydoop
- 53:09Reading HDFS files in Pydoop
- 57:10Python MapReduce performance
- 1:01:20Pydoop real-world use cases
- 1:04:36Summary and acknowledgments
related talks
44mBig Data with Python & Hadoop
50mBig Data Processing and Data Science Environment in Python
44mUsing Python with Hadoop to create production ready big data applications
01h 01mParallel Python Analyzing Large Data Sets
25mAutomated Scientific Pipelines With Doit
41mMassively Parallel Processing with Procedural Python
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
40mScaling Python - Bank Edition