Python and the JVM - A Love Story
summary
Claims that the JVM gives 10× faster runtime for some workloads and that Py4J lets Python call JVM objects via a Py4J gateway server. Shows Scala code exposing an entry-point object, then a Python FastAPI that submits ad hoc Spark jobs to an on-demand Spark simulator backed by real Spark clusters. Demonstrates running this in production at Hunters, explains classpath and Scala module quirks, and shows the remote-client option where Python stays local and heavy Spark runs on a powerful JVM server.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:12Combine Python and JVM
- 4:41Setting up Py4J for JVM-Python communication
- 9:02Py4J bridging runtimes
- 11:28Apache Spark use case
- 18:14Expose JVM objects
- 20:41Q&A
related talks
33mIntegrating Scala Java with your Python code
56mJourney from Java to Python: Two Worlds, One JEP
31mBringing the Jewels of the Python World to Scala with Spark
41mKeynote: Making the Big Data ecosystem work together with Python
47mSupercharge your Java Applications with Python!
41mImproving PySpark Performance Spark performance beyond the JVM
01h 00mPython and Julia: the best of friends?
46mJulia, Python, R & Scala – A Comparison From Experience