How We're Democratizing Stream Processing at Zillow
summary
Zillow replaces slow ticketed builds with a self‑service platform that provisions Flink jobs, Kafka topics and permissions in minutes using Kubernetes as the control plane. It reuses Lyft’s Flink operator, implements Kafka controllers with the Kubernetes metacontroller, and exposes everything via a custom Terraform provider plus Helm charts and a CLI. CI/CD pipelines and Datadog dashboards are auto‑generated; demo shows end‑to‑end deploys and monitoring working reliably, cutting per‑project onboarding and ops overhead dramatically.
AI-written summary · about
chapters (17)
- 0:00Introduction to Zillow
- 2:14Stream processing at Zillow
- 5:22Challenges in stream processing
- 6:23Why build a platform
- 7:26Platform requirements
- 11:18Kubernetes as foundation
- 15:26Flink operator adoption
- 18:56Kafka operators development
- 21:05Terraform integration
- 24:14CI/CD and monitoring
- 26:21Documentation and branding
- 27:45Versioning and release notes
- 28:49CLI tool for engineers
- 30:35Demo: Project initialization
- 35:05Demo: Kafka topics provisioning
- 36:31Demo: Flink job deployment
- 38:14Conclusion and Q&A
related talks
38mInteractive Session: Flink on Autopilot with Splunk
22mI²: Interactive Real-Time Visualization for Streaming Data with Flink & Zeppelin
27mHow to be data-driven when you aren’t Netflix (or even if you are)
44mApache Zeppelin: A friendlier way to Flink
30mNRTG: Fast, Systematic & Scalable Flink Application Development at Pinterest
06mLightning Talk: Blueprints of Innovation: Engineering Paved Paths for a User...
38mQuest to Connect Teams with Kafka
30mTeam Teardown: Driving Platform Adoption and Success