KalDB: A cloud native log search platform
summary
Replaces Elasticsearch with a lucene based cloud-native store that prioritizes live data over backlog, uses Kafka partitions and schema-on-read to avoid field-conflict data loss, and snapshots indexes to S3 backed storage so older data lives cheaply in deep store. Architecture separates indexer/cache/query with recovery indexers for spikes; currently runs in dark mode indexing roughly 7 million messages per second and managing about a petabyte with seven-day retention, claiming up to 10x cost reduction and much lower operational toil.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:34About me
- 1:33Quiz
- 6:58Log Types
- 14:02Architecture
- 17:34Motivation
- 18:24KalDB Architecture
- 21:39Multitenancy
- 23:34Delayed logs
- 25:48Calib
- 27:55Field Conflicts
- 29:07Cost Savings
- 32:39Conclusion
related talks
07mKalDB: Log analytics with DuckDB
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
34mFoundationDB, the Black Knight
14mA Cloud-native Architecture for Replicated Data Services
43mWhy We Built Our Own Distributed Column Store
01h 02mRonDB: A Key-Value Store with SQL Capabilities and LATS Properties
52mData storage for modern applications - Mike Lehan
26mBuilding Data Lake Using DuckDB