Using serverless functions for real-time observability

on YouTube ↗49m

" summary

A custom retriever column store (ClickHouse-like) moves older shards to S3 and farms queries to AWS Lambda + S3 workers so interactive queries cover months of data; the system reaches petabyte scale with ~50k prewarmed workers, P90 ≈ 2.5s, and retrying slow S3 reads. Key tradeoffs: Lambda is ~3–4×/ms vs EC2 but wins on utilization, payload and request costs matter, and arm64 cost saving ~20–30% requires architecture builds, profiling and batching.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to serverless
2. 1:08Defining the use case
3. 3:26Observability and flow
4. 7:22Architecting for scale
5. 11:51Tearing and storage strategy
6. 15:49Serverless for query compute
7. 18:41Scaling and concurrency limits
8. 22:40Cold starts and optimization
9. 28:11Cost management
10. 31:50Migrating to ARM64
11. 35:51Performance engineering
12. 41:14Conclusion and best practices

" related talks

• 41mUsing Serverless & ARM64 for Real-Time ObservabilityLiz Fong Jones · GOTO Chicago 2024
• 32mUsing Serverless Functions for Real-time ObservabilityLiz Fong Jones, Jessica Kerr · SREcon22 Americas
• 36mHow we used serverless to speed up our serversJessica Kerr, Ian Wilkes · Strange Loop 2021
• 39mServerless in production, an experience reportYan Cui · Codemotion Milan 2017
• 47mServerless in production, an experience reportYan Cui · Build Stuff 2017
• 01h 00mServerless Architecture - Tales from a world without serversRobin Weston · NDC London 2017
• 01h 02mServerless in production, an experience reportYan Cui, Domas Lasauskas · NDC London 2018
• 29mHow to build a social network on serverlessYan Cui · apidays Barcelona 2019