Python, Java, or Go: It's Your Choice ...
summary
Apache Beam exposes a portable pipeline model that lets Python, Java, Go and Scala run on engines like Apache Flink and Google Cloud Dataflow via a job server plus a FN API contract; user code runs inside Dockerized SDK harness runtimes and the runner performs pipeline fusion to reduce container overhead. Cross-language transforms let a Python pipeline call existing Java IO connectors (Kafka, S3), avoiding years of reimplementations. Expected overhead from the portability layer is roughly 10–15% while Java and Python SDKs are production-ready and Go is mature for batch.
AI-written summary · about
chapters (5)
- 0:00Introduction to Apache Beam
- 9:24Beam API and transforms
- 12:54Language portability framework
- 27:59Current portability status
- 32:39Q&A session
related talks
10mDemocratizing the pipeline with Apache Beam and Dataflow
29mApache Beam Making Big Data Portable With gRPC
31mMaking Big Data Processing Portable. The Story of Apache Beam and gRPC - Ismaël Mejía
41mApache Beam: Unified Data Processing
47mApache Beam - what do I gain?
21mApache Beam (incubating)
39mApache Beam and Google Cloud DataFlow
26mBeam and dataflow: a perfect combination of parallel processing