Postmortem: Evolution of an Event recording solution
summary
Company needs per-user event logs so a lightweight on-host system is built: Tornado web processes spawn a multiprocessing ZeroMQ pub/sub worker that batches messages and performs batched INSERTs to Postgres every second. The design avoids Kinesis/Lambda and external vendors, records GET/POST, click payloads, user and org IDs, and logs status codes. Durability is best-effort (no broker or disk queue), which is acceptable for analytics, and a quick test handles 10,000 events in 1.5s on one box.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:19About Me
- 1:02Problem Statement
- 1:38Won't work for us
- 3:15Our Environment
- 4:05Commercial Solutions
- 5:33Log Parsing
- 6:37Bad Idea
- 7:18Our First Design
- 8:12How to implement
- 10:48How does Python Triton Work
- 11:14Wait a minute... Zero MQ?
- 13:05How does ZeroMQ work?
- 17:30Let's build a simpler solution
- 23:59The Client
- 25:36Start the Worker Process
- 26:36Talk about scalability
- 28:34Conclusions
- 31:58Questions?
related talks
29mWide Event Analytics
27m"Write Once Stream Everywhere: Transforming Your Data into Events"
44mPolyglot Data
57mLessons in Building a System that Processes More than 70 Billion Events Daily
51mImmutable data stores and CORS for safety, flexibility and profit
40mHow to Process 80 Million Events a Day and Build Relational Graphs with Postgres
43mUsers As Data
24mEvent Analytics - from unstructured to insightful