A Quick Primer on TensorFrames: Apache Spark and TensorFlow Together

on YouTube ↗01h 56m

" summary

TensorFrames claims to let Spark and TensorFlow exchange tensors efficiently — TensorFrames lets Spark DataFrame talk to TensorFlow without pickling by using Spark’s binary format — and shows concrete speedups versus UDFs and Scala-only code, with optional GPU acceleration. Demos run on Databricks (Spark 2.1, Scala 2.11) and include MNIST: MNIST demo: softmax ~91% then CNN 98.95%. Notebooks and install steps are provided: Databricks + Spark 2.1 (Scala 2.11); install tensorflow and tensorframes (CPU demo; GPUs give more speed).

AI-written summary · about

" chapters (11)

1. 0:00Introduction to TensorFrames and speakers
2. 6:07Neural networks basics
3. 21:51TensorFlow overview
4. 27:09Why TensorFrames
5. 30:23Setting up Databricks Community Edition
6. 48:10Installing libraries
7. 49:12MNIST dataset
8. 1:01:20Softmax regression
9. 1:16:08Building a convolutional neural network
10. 1:32:00TensorFrames with Spark
11. 1:48:10Q&A

" related talks

• 34mWorking with Tensorflow from the JVM #bbuzzHolden Karau · Berlin Buzzwords 2018
• 52mPractical Uses of Scala Tensorflow for Image AnalysisXavier Tordoir · ScalaIO 2019
• 22mBringing Deep Learning into the Big Data AnalyticsYuhao Yang · Scale By the Bay 2017
• 36mHigh-Performance Distributed TensorflowChris Fregly · PyData London 2017
• 30mSpeed Up Your Spark Data Processing with a GPURaphaël Luta · Devoxx France 2021
• 45mTensorly: A Flexible Python Framework for Machine LearningAnima Anandkumar · PyData Los Angeles 2018
• 20mTensorFlow.js Bringing Machine Learning to the Web and BeyondNick Kreeger, Nikhil Thorat · JSConf EU 2019
• 36mScaling data workloads using the best of both worlds: pandas and SparkEng & Kwon · PyData Seattle 2023