Big Data with Python & Hadoop
summary
Hands-on comparison of Python tooling for Hadoop: HDFS access with the native CLI versus snakebite as the fastest HDFS client, job wiring via Hadoop Streaming, Python frameworks (mrjob, Luigi, Pydoop/pipes, Pig UDFs with Jython), and interpreter tradeoffs (CPython vs PyPy). Benchmarks run a wordcount on Learning Python scaled to 35 GB showing Java baseline 187 s and Pydoop ~1.86× slower; Luigi and default serialisation perform much worse. Slides and code links are provided.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:29Agenda
- 3:10What Exactly Is Big Data
- 4:20Apache Hadoop
- 9:20Mapreduce
- 10:50Example of Mapreduce
- 13:16Hadoop Streaming
- 13:48Code for a Mapper and Reducer
- 16:09Hadoop Distributed Cache
- 19:46Mapreduce Job
- 21:00Install Simple Json
- 23:59Luigi
- 25:45Benchmarks
- 27:27Performance Comparison
- 34:28Pros and Cons
- 42:25Conclusion
related talks
01h 05mPython MapReduce Programming with Pydoop
44mUsing Python with Hadoop to create production ready big data applications
01h 23mMapReduce Programming in Python with Pydoop
01h 22mIntroduction to Big Data Processing using Spark and Python
41mKeynote: Making the Big Data ecosystem work together with Python
50mBig Data Processing and Data Science Environment in Python
01h 01mParallel Python Analyzing Large Data Sets
01h 10mGetting started with H2O on Python