Transforming Analytical Data with Event-Driven Architecture
summary
Argues that analytical data must be created as a first-class product, not recycled from operational DBs, and demonstrates a reproducible stack: domain-driven data contracts defined in Protobuf (or AsyncAPI/JSON Schema), registered in a schema registry, emitted to Kafka, validated and enriched by Spark Streaming, and landed to S3 as Parquet/Iceberg tables. Shows automated SDK generation to force instrumentation in client/server code, claims Wix runs this across ~2,000 microservices with ~200 contracts and processes billions of events daily, enabling near real-time pipelines and automated ingestion/validation for reliable analytics.
AI-written summary · about
chapters (17)
- 0:00Introduction and welcome
- 1:09About the speaker and Wix
- 2:12Analytical vs operational data planes
- 3:38Why companies need data-driven decisions
- 7:13Problems with current data recycling approach
- 11:44The data recycling center analogy
- 15:56Shifting left data responsibility
- 16:39Introducing data contracts
- 20:18Domain events as data contracts
- 23:29Implementing service contracts at Wix
- 28:23Automating data consumption with contracts
- 30:51Real-time analytics example
- 34:45Second strategy: Analytical data instrumentation
- 37:52Instrumenting data from business needs
- 41:44Architecture for instrumentation
- 43:11Benefits of data contracts
- 44:56Future of data platforms and Meetup
related talks
34mTales of Data Architecture Evolution
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers
37mBuilding Event-driven Apps at Scale
31mOne Data Pipeline to Rule Them All
18mBehavioral Data Creation for AI - Delivering Value and Insights
30mPlatform Oriented Data Architecture for Acceler...
31mDemystifying Event-Driven Architectures
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications