Your Data Pipeline with Apache Beam
summary
Shows how to build a payments data pipeline using Apache Beam that reads from Pub/Sub and writes to BigQuery, with example Java code and notes about portability to Python, Go and SQL and runners like Dataflow, Spark or Flink. Implements per-recipient aggregation using a 30-minute sliding window every 5 minutes with a trigger that fires on 100 elements or 1 minute to emit intermediate results, and discusses event time vs processing time and Beam’s built-in connectors. Reports that Beam works in practice but has a learning curve, limited community support and gaps in testing.
AI-written summary · about
chapters (6)
related talks
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
41mApache Beam: Unified Data Processing
39mApache Beam and Google Cloud DataFlow
47mApache Beam - what do I gain?
52mApache Beam, a unified model for Big Data processing
21mApache Beam (incubating)
26mBeam and dataflow: a perfect combination of parallel processing
10mDemocratizing the pipeline with Apache Beam and Dataflow