Blurring the line between Developer and Data scientist with PixieDust
summary
Claim: PixieDust makes Jupyter+Spark let developers and data scientists work the same way by turning dataframes into instant apps. It ships as an open source Python library with display(df) auto-visualization, real‑time Spark job status, a Scala bridge / package manager to import jars, and embedded-app templates. Demo connects Spark Streaming to Twitter, calls Watson Tone Analyzer, runs 30 seconds, ingests ~67 tweets into a Spark DataFrame, and exposes an interactive dashboard for nontechnical users.
AI-written summary · about
chapters (19)
- 0:00Start
- 0:34Why are you here?
- 2:44Developers are the "New Builders"
- 4:51Meet Ben "the developer"
- 5:34Meet Natasha "the data scientist"
- 6:09Surprise meeting with VP of development
- 6:42Key Constraints
- 7:29What is Data Science?
- 8:49Some learning to do
- 9:49Spark Core Libraries
- 11:53Consuming Spark
- 12:49Ben and Natasha start brainstorming
- 14:01Can we collaborate using Notebooks?
- 14:23What is a Notebook
- 15:33What is Jupyter?
- 16:42Big Data Analysis
- 17:38Notebooks are powerful tools for data scientists
- 18:31Pixie Dust
- 19:08Demo
related talks
37mPixieDust make Jupyter Notebooks with Apache Spark Faster, Flexible, and Easier to use
45mIndroduction to the PySpark DataFrame API
44mUsing Python with Hadoop to create production ready big data applications
24mPySpark - Data processing in Python on top of Apache Spark.
31mBringing the Jewels of the Python World to Scala with Spark
50mBig Data Processing and Data Science Environment in Python
47mScala: The Unpredicted Lingua Franca for Data Science
41mKeynote: Making the Big Data ecosystem work together with Python