Apache Beam Making Big Data Portable With gRPC
summary
Apache Beam implements a portability framework that uses gRPC-based Fn API plus Job, Artifact, Provision, Data, State and Logging services to run user functions in sidecar language containers, decoupling language from runner. Java, Python and Go SDKs translate pipelines into protobuf job specs; runners include Flink, Spark and experimental Google Dataflow and Samsung support. Validation uses the Validated Runner suite TCK; outcome shows isolated dependencies, cross-language pipelines (e.g. TFX on open-source runners) and minimal gRPC overhead at large scale.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:20Distributed Data Processing (aka Big Data)
- 6:02The Apache Beam Vision
- 6:39Beam Programming Model
- 7:15How do Java-based runners work?
- 8:18How JVM systems support Python (Spark)
- 10:04Portability API Design
- 11:16Portability APIs Well-defined, language-nestral data structures and protocols between the SDK Harness and runner
- 20:27Advantages of the Portability Framework Isolation of user code Isolated configuration of user environment A blueprint for new languages support Better architecture more code reuse between runners
- 22:31Portable Capability Matrix
- 22:55Quality Assurance
- 23:36Nice Outcome: TFX on Open Source systems Open source end-to-end platform for deploying production ML pipelines Several of the TFXibraries use Beam (Python) for running tasks
related talks
31mMaking Big Data Processing Portable. The Story of Apache Beam and gRPC - Ismaël Mejía
41mApache Beam: Unified Data Processing
47mApache Beam - what do I gain?
21mApache Beam (incubating)
10mDemocratizing the pipeline with Apache Beam and Dataflow
41mPython, Java, or Go: It's Your Choice ...
52mApache Beam, a unified model for Big Data processing
46mBuilding an Apache Beam streaming runtime in GCP