Using Python with Hadoop to create production ready big data applications
summary
Shows concrete pipelines: PySpark + MLlib ALS recommender on MovieLens deployed with spark-submit, backed by Spark as the data store and cached RDDs for repeated queries. Demonstrates Zeppelin notebooks with interactive Bokeh HTML outputs and D3/json export and how those single-file visualizations get embedded and shared. Presents a Flask API + CherryPy/WSGI front end that accepts browser queries and returns JSON from Spark, plus the exact commands, file layout and runtime tradeoffs for memory, cores and latency.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 5:39Zeppelin notebook overview
- 12:16Zeppelin and libraries
- 22:07Spark recommendation engine
- 31:54Flask and CherryPy web app
- 39:07API for recommendation system
- 41:49Q&A
related talks
01h 22mIntroduction to Big Data Processing using Spark and Python
50mBig Data Processing and Data Science Environment in Python
41mKeynote: Making the Big Data ecosystem work together with Python
39mHow Python Drives Modern Data Engineering
31mBringing the Jewels of the Python World to Scala with Spark
24mPySpark - Data processing in Python on top of Apache Spark.
37mPixieDust make Jupyter Notebooks with Apache Spark Faster, Flexible, and Easier to use
44mBig Data with Python & Hadoop