Skein: a simpler way to deploy applications on Hadoop Clusters

on YouTube ↗26m

" summary

Skein is a small library and CLI that actually runs Python apps on Apache YARN by shipping a Java application master that speaks gRPC to Python clients and manages containers; it includes Conda pack and VM pack for moving full environments, an embedded key/value store for coordination, an extensible web UI, and a testing tool called Hadoop test cluster. Demos show scaling, logs, and a working desk yarn integration; teams run TensorFlow on YARN and prototype Ray and Jupyter kernels with it.

AI-written summary · about

" chapters (15)

1. 0:00Skein and Hadoop clusters
2. 2:53Skein in action
3. 4:18Deploying Python on YARN
4. 6:00Skein’s declarative spec
5. 7:53Skein and YARN clusters
6. 9:20Key features of Skein
7. 11:03Python environments for YARN
8. 13:13Conda with PySpark
9. 14:38Skein’s key-value store
10. 16:46Skein’s web UI
11. 18:10Testing YARN apps
12. 20:21Desk Yarn for Python
13. 21:26Skein future work
14. 22:49Summary of Skein
15. 23:31Q&A: Conda Pack with Hadoop

" related talks

• 33mIntegrating Scala Java with your Python codeMarius van Niekerk · PyData Carolinas 2016
• 43mPySpark in PracticeDat Tran, Ronert Obst · PyData London 2016
• 44mUsing Python with Hadoop to create production ready big data applicationsUlrich Zink · PyData London 2016
• 26mDeploy Python Apps on Kubernetes and Prepare for ScaleSenthil Kumaran · PyBay 2024
• 31mBringing the Jewels of the Python World to Scala with SparkHolden Karau · Lambda World Seattle 2018
• 25mDeploying Python at Scale with DaskMatthew Rocklin · PyCon US 2020
• 32mMachine Learning Infrastructure at Stripe: Bridging from Python JVMRob Story · PyData Seattle 2017
• 31mAdding Python support to Scala toolingAlexandre Archambault · ScalaIO 2022