Kaldb: serverless lucene at petabyte scale
summary
An open-source Lucene serving system claims to run petabyte-scale logs on Kubernetes by separating compute, durability and storage: Kafka acts as a write-ahead log, indexers pull partitions, snapshots go to S3 and cache nodes hydrate on demand. It reports running ~1 PB with seven-day retention, peak ingest ~70 Mbps and ~7M msgs/sec, handles field-type conflicts automatically, spins ephemeral recovery indexer tasks to catch up on spikes, uses disaggregated storage and Kafka as WAL, and says costs can be up to 10x lower.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 2:05Motivation
- 3:01Research
- 4:28Conclusion
- 5:27Distributed database design
- 6:07generative search
- 7:11history of column stores
- 8:16serverless architecture
- 9:26Kaldb summary
- 10:55Kaldb
- 12:20Challenges
- 12:41High operational load
- 14:38Kaldb components
- 16:05Cluster Manager
- 16:49disaggregated storage architecture
- 19:21handling log spikes
- 22:09quiz question
- 22:25elasticsearch
- 25:24recovery indexer
- 26:16multitenancy
- 27:32workload isolation
- 28:54data reliability
- 31:15field conflict resolution
- 32:06cost
- 34:49summary
- 35:32questions
related talks
32mBuilding and scaling a log analytics platform - a serverless approach
56mScaling Telemetry Systems with Streaming & Serverless
49mUsing serverless functions for real-time observability
01h 02mServerless in production, an experience report
26mBlazing-Fast Serverless MapReduce Indexer for Apache Solr
39mServerless in production, an experience report
36mLeveraging Serverless architecture with Confluent Kafka
33mNext Gen of Datastores: Hot Queries, Cold Storage