Data governance in streaming at scale
summary
Letgo builds a company-wide data model using Avro schemas with field-level privacy tags that block or redact nonconforming events, and runs ingestion on Kafka Connect into a tiered AWS data lake; Spark performs deterministic transforms and lake ingestion. The platform enforces schema and quality gates for both real-time and batch, maps datasets into a multi-tiered data lake (inner tier for sensitive records, outer tiers per business unit), and claims reliable, low-maintenance governance at scale.
AI-written summary · about
chapters (7)
- 0:00Introduction to data governance
- 5:24Access control and privacy issues
- 9:00Decentralized data governance
- 13:31Repository and data sharing
- 19:00Data processing and collection
- 23:13Real-time data services
- 25:54Summary and closing
related talks
31mOne Data Pipeline to Rule Them All
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
30mBeyond the Build: Unleashing the Hidden Value in Your Data Platforms
39mCrafting modern data platforms: reveal your data value
29mGrowing insights through hybrid data architectures
05mInterview