Building domain specific databases with a distributed commit log
summary
Smyte replaces Redis with Kafka + RocksDB, writing timestamped increments to Kafka and storing count plus Kafka offset in the same RocksDB transaction to get exactly-once semantics. Sliding-window counts move from 15-minute buckets to second-level accuracy; consumers decrement delayed messages for hour/day/week windows and older segments roll to S3. Replication uses persistent disks via Kubernetes so replicas mount the same disk; a new replica catches up by consuming ~200k messages per partition. Costs drop by avoiding in-RAM Redis.
AI-written summary · about
chapters (11)
- 0:00Introduction to Smite and counting
- 3:25Version 0 with Redis
- 5:58Moving to Kafka for durability
- 8:52Cost and fault tolerance challenges
- 10:38Requirements for version 3
- 12:25Building a custom database
- 14:14Replication with Kubernetes
- 17:30Benchmarking and scaling
- 18:13Final architecture and framework
- 21:23Future work and unique counts
- 22:47Q&A session
related talks
33mBuilding domain specific databases using Python for prototypes during take off
36mTime-based domain events - how to deal with time
43mWhy We Built Our Own Distributed Column Store
33mThe rise and fall of the full stack database
57mRocksDB
01h 19mBuilding an Efficient Replication System for Thousands of Terminals
36mBuild a Cloud Native Asynchronous Messaging System For Scale With Redis
30mData @Scale Seattle- Muthu Annamalai