Turning the database inside out with Apache Samza

on YouTube ↗48m

" summary

Claim: turn the database inside out by making the immutable commit log the primary data model and building materialized views from it. Implementation uses Apache Kafka as a distributed durable log and Apache Samza to stream-process and maintain per-partition local state. Result: fully precomputed caches (no cold starts), simpler, sequential update semantics, better scalability and recoverability. Tradeoffs: views update asynchronously so clients see eventual consistency; idempotence and transactional layers are needed for exact once semantics.

AI-written summary · about

" chapters (10)

1. 0:00Database architecture shift
2. 3:55Shared mutable state problems
3. 7:50Immutability in replication logs
4. 11:05Flaws in app caching
5. 14:19Materialized views limitations
6. 17:49Four forms of derived data
7. 20:59Turning databases inside out
8. 25:12Log-centric architecture benefits
9. 29:46Streams in app design
10. 35:43Q&A

" related talks

• 43mBack to fundamentals: from databases to event-driven platformsFlorent Ramière · DataXDay 2020
• 34mChasing the Stream Processing UtopiaKartik Paramasivam · Strange Loop 2018
• 20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™Celeste Horgan · Berlin Buzzwords 2026
• 48mUnlocking Real-time Insights: Enhancing Your Databases With Stream ProcessingYingjun Wu · Big Data Conference Europe 2023
• 33mThe rise and fall of the full stack databaseFrank Lyaruu · Full Stack Fest 2017
• 44mHow to Manage State in Apache KafkaBen Abramson · JAX London 2019
• 17mOvercome By Events - Apache Kafka And Event-oriented ArchitectureJay Kreps · Big Data LDN 2018
• 39mModern Event-driven Architecture: Adopting Data in MotionAlex Stuart · GOTO EDA Day 2022