Using Serverless Functions for Real-time Observability
summary
A sub-second query engine at honeycomb.io moves from RAM to local SSD to months of S3-backed segments, then parallelizes heavy queries by spawning 30,000 concurrent lambdas that each process shards and return partial aggregates. Real issues include account-level concurrency 429s, cold-start and S3 tail latency mitigated by impatience retries and lambda layers, payload limits like 6 MB return limit and large file lists, and surprising cost spikes (internal SLO testing blew a multi‑thousand dollar bill). Fixes: lz4 compression, Go runtime and arm64 tuning, feature flags, and coordinating with the cloud account team.
AI-written summary · about
chapters (12)
- 0:00The problem of latency
- 1:06Observability and retriever
- 2:23Interactive investigation
- 4:22Designing a fast data store
- 7:47Scaling with S3
- 9:33Adding serverless compute
- 12:06Realities of serverless
- 16:54Handling tail latency
- 19:28Managing costs at scale
- 22:10Optimization experiments
- 24:52Lessons from implementation
- 28:12Best practices for serverless
related talks
36mHow we used serverless to speed up our servers
49mUsing serverless functions for real-time observability
41mUsing Serverless & ARM64 for Real-Time Observability
23mBuilding & Running a Scuba-Inspired Observability Tool
54mServerless beyond Functions
01h 01mServerless at scale
44mServerless Architecture on Production: Good, Bad and Ugly Parts
39mServerless in production, an experience report