Data Snarfing with Go: Heka Good Time
summary
Heka is a high performance data collection and processing system that actually runs in production it uses protocol buffers for a shared message format, Go goroutines and channels for plugin concurrency, and Lua sandboxes for dynamic, fast text decoding with LPEG grammars The system hits roughly 400,000 messages/second on a laptop, handles dynamic filter loading without restart, but currently lacks strong end to end delivery guarantees and has some cgo static linking and cross platform build pain points
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:17Mozilla Cloud Services challenges
- 3:24Data processing patterns
- 5:50Heka design and real-time
- 8:42Go plugins and concurrency
- 11:34Message routing and Lua sandbox
- 13:25Why Lua for parsing
- 16:15LPeg parsing power
- 19:02Go chosen for Heka
- 22:10Go performance benefits
- 23:37Go static linking issues
- 26:50Go gotchas and pitfalls
- 28:15Concurrency safety
- 29:05Heka in production
- 31:26Data loss risks
related talks
39mGophers riding Elephants
28mManaging and Scaling Real Time Data Pipelines using Go
34mGoHEP: what can Go do for science?
57mHacking with Go
31mHow Go is Making us Faster
28m(Neural) Natural Language Processing in Go
30mDynamically Instrumenting Go Programs
21mMy Protobuf Module is Faster than Yours (Because I Cheated)