Building & Running a Scuba-Inspired Observability Tool
summary
Honeycomb builds Retriever, a columnar datastore that pairs NVMe-local shards and Kafka ingest with S3 tiering and tens of thousands of concurrent Lambdas for parallel leaf aggregation, claiming interactive read-time queries in roughly 100 milliseconds to one second while supporting petabyte scale compressed storage and up to 60 days of retention. Implementation details: consistent hashing to co-locate trace spans, leader+local retriever two-tier aggregation, S3 cold-path with retries to mask long tails; this design runs in production and meets the latency and durability targets.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:41What is Honeycomb
- 1:55What is Retriever
- 9:34Speed and Scale
- 13:17tradeoffs
- 17:47durability
- 19:12verification
- 21:25features
related talks
32mUsing Serverless Functions for Real-time Observability
43mWhy We Built Our Own Distributed Column Store
49mUsing serverless functions for real-time observability
33mLeveraging SRE and Observability Techniques for the Wild World of Building on LLMs
20mBuilding An Elastic Query Engine on Disaggregated Storage
31mBuilding large scale, transactional data lakes with Apache Hudi
40mInsights into Your Cloud Database: How Storage Engines Actually Work
40mObservability that scales: Analyze data in stream and query remotely