Data Integration at SafetyCulture: Implementing Protobuf for Efficient Data Contracts
summary
SafetyCulture replaces many bespoke CDC feeds with a single central CC Kafka topic and Protobuf contracts that enforce an envelope with metadata (who, when, org, UUID). Message validation blocks malformed records before downstream processing, dropping alerts and failures to nearly zero. Pipeline onboarding time falls from days to half a day—claimed as 90% faster onboarding—and adoption reaches 100% across teams. The stack uses automated notifications and gateways so schema changes require collaborative review, not emergency firefighting.
AI-written summary · about
chapters (8)
- 0:00Data integration journey
- 1:13Safety Culture platform
- 2:59Original data architecture problems
- 8:04Protobuf for standardization
- 10:56Automating pipelines and governance
- 13:02Team culture change
- 14:49Protobuf results and impact
- 17:17Q&A
related talks
43mBuilding Scalable and Extendable Data Pipeline
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
33mAccidental DataOps: How we matured our organization with data, and how you can too!
27mLive in the Data Wild West: The Data Contracts Sheriff
29mUnifying Real-Time and Data Lake: Yotpo's Transition from Chaos to Coherence
31mOne Data Pipeline to Rule Them All
26mData contracts: driving data culture change
49mTransforming Analytical Data with Event-Driven Architecture