KalDB: Log analytics with DuckDB
summary
KalDB adds a plugable storage engine so one logging system serves both search and analytics by combining Lucene for search with DuckDB for fast analytics; it runs on Kubernetes, ingests from Kafka, snapshots indexes to S3, and serves queries via a scatter‑gather cache layer. Claims: petabyte scale operation with about 7 million msgs/sec peak and seven‑day retention. Current issues: mapping Lucene semantics to DuckDB, defining schemas for schemaless logs, and sharp ingestion throughput drops as column count grows, plus response‑format translation for distributed aggregation.
AI-written summary · about
related talks
40mKalDB: A cloud native log search platform
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
28mAnalyze data easily with duckdb - and the implications on data architectures
26mBuilding Data Lake Using DuckDB
51mMotherduck: Data Analytics in the Post Big-Data Era
40mDuckDB: Harnessing in-process analytics for data science and beyond
01h 02mAnalytics for not-so-big data with DuckDB
45mAnalytics for not so big data with DuckDB