Streaming data processing pipelines with Apache Beam [in Python, naturally!]
summary
Covers using Apache Beam (with the Python SDK) to build streaming data processing pipelines for real-time analysis and action. Explains why batch processing is insufficient and demonstrates unified data processing on Google Cloud Platform technologies.
AI-written summary · about
chapters (20)
- 0:00Start
- 0:50Google research in data technologies
- 2:24Apache Beam & Cloud Dataflow
- 3:52What is part of Apache Beam? One Model Multiple Modes Multiple SOK
- 4:46Customizing What Where When How
- 8:11Installation
- 9:16Functioning Hello Beam Code
- 14:03Beam Pipelines are Directed Acyclic Graphs (DAG)
- 18:29General Structure
- 20:36Data and Functions a.k.a PCollections and PTransforms
- 22:10Getting Data with
- 24:26Piping Transforms
- 26:23Parallel, Distributed Processing - Pardo
- 27:22Pardo- Parallel Do
- 30:40Branching the Pipeline
- 31:18Windowing
- 33:01Writing Output to Data Store
- 33:31Logging
- 33:54Local Testing and Cloud Production
- 35:21Point Of Sale (POS) Enrich Streaming Fact Data with Dimension Table Data
related talks
38mData processing with Apache Beam
16mYour Data Pipeline with Apache Beam
24mApache beam and google cloud dataflow
52mApache Beam, a unified model for Big Data processing
10mDemocratizing the pipeline with Apache Beam and Dataflow
39mApache Beam and Google Cloud DataFlow
26mBeam and dataflow: a perfect combination of parallel processing
21mApache Beam (incubating)