Making Big Data Processing Portable. The Story of Apache Beam and gRPC - Ismaël Mejía
summary
Apache Beam implements a portability layer using protobuf + gRPC services (Job API, FN API, control/data/state APIs) and containerized SDK harnesses to run user code in any language, with artifact staging and provision services for dependencies. A universal local runner exists and a Python wordcount on Flink runs end-to-end; early tests show about 15% overhead. The system is functional for experiments but not production-ready yet, with more SDKs (Go) and full model coverage planned by Q4.
AI-written summary · about
chapters (26)
- 0:00Intro
- 1:04New products
- 1:29Apache Beam origin
- 2:20What is Apache Beam?
- 2:38Beam Model: Generations Beyond MapReduce
- 3:01Processing Time vs. Event Time
- 4:30The Beam Model: What is Being Computed?
- 5:09The Beam Model: Where in Event Time?
- 5:58The Beam Model: When in Processing Time?
- 6:45The Beam Model: How Do Refinements Relate?
- 7:25Customizing What Where When How
- 7:39Beam Pipeline
- 8:10Apache Beam - Programming Model
- 8:40The Apache Beam Vision
- 11:15How do Java-based runners do work today?
- 12:07Portability API Design
- 15:06Portability Framework
- 15:35Runner API
- 16:15Job API
- 21:24FnAPI-Control - Bundle Processing
- 22:29Advantages
- 23:55Current status
- 24:41Go SDK
- 25:22Ongoing/Future work
- 28:34Greetings
- 28:46References
related talks
29mApache Beam Making Big Data Portable With gRPC
41mApache Beam: Unified Data Processing
47mApache Beam - what do I gain?
10mDemocratizing the pipeline with Apache Beam and Dataflow
21mApache Beam (incubating)
52mApache Beam, a unified model for Big Data processing
41mPython, Java, or Go: It's Your Choice ...
46mBuilding an Apache Beam streaming runtime in GCP