Data @Scale Seattle- Sundaram Narayanan
summary
Built LogLens, a lightweight log service replacing Elasticsearch with a pipeline using Kafka into partitioners, ~100–150 one‑core indexers and HDFS for archives; ingestion hits ~250–350 Mbps and 300k–400k docs/sec, cache holds ~20–50 TB across small cache nodes, and most restores complete in ~5 minutes. Design uses stateless partitioners, a retention manager, and compact indexers; ~10k lines of code, ~4 months work, trades deep analytics for predictable cost control and tiered availability.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:47Agenda
- 1:31Stateless Services
- 4:20Aggregation Search Analytics
- 6:00Problem Characteristics
- 7:00The Hack
- 9:06Requirements
- 11:40Tiered Availability
- 14:07Onboarding
- 15:24Design
- 17:00Model
- 21:30Rate Control
- 24:03Future
- 26:00Questions
related talks
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
31mLogging at Scale, Billion+ messages, 100K msg/sec
46mBuilding a scalable logging platform
40mLarge-Scale Logging Made Easy
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
40mObservability that scales: Analyze data in stream and query remotely
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution