Using Open Source Tech to Swap Out Components of Your Data Pipeline
summary
Public bike availability is polled from a public API, ingested and transformed with Apache Beam, and run locally or on cloud with a simple runner switch to Dataflow runner. The live project collects ~5 GB of raw JSON over a year, produces ~13 million rows (~1 GB) in BigQuery, and reruns in ~14 minutes by scaling to ~160 workers; it works but breaks when the provider changes the API (pretty‑printed JSON) so expect data‑wrangling.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 7:51About Rustam
- 13:20Pipeline definition
- 14:19Data overview
- 15:26tabular form
- 16:15parallel processing
- 16:47Paralysis
- 17:52File Structure
- 18:58Local Runner
- 20:33Cloud Runner
- 22:26Data Flow
- 28:29File System
- 33:58Build Success
- 41:43Takeaways
related talks
46mUsing Open Source Tech to Swap Out Components of Your Data Pipeline
16mYour Data Pipeline with Apache Beam
26mETL pipeline with GCP Dataflow
41mGoogle Dataflow: New open model for batch & stream processing
47mApache Beam - what do I gain?
46mBuilding an Apache Beam streaming runtime in GCP
21mPredicting Hacker News with Beam and TensorFlow
44mImprove your App UX with one model processing insights from batch and streams