Use Python to process 12mil events per minute and still keep it simple (Talk)
summary
Claims handling 12 million requests per minute using small four‑core AWS instances, built around reads nginx access log as a durable queue, and implemented in Python 3 and Cython with multiprocessing queues and simple on‑disk binary file offsets and a binlog for exactly‑once recovery. Measured throughput hits about 20k req/s per instance, cluster of ~15 VMs serves peaks, Cython doubles single‑node performance, fsync on AWS EBS causes rare 0.1–25s stalls that must be mitigated for safe persistence.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:27What is an advanced
- 1:14The problem
- 1:29What is 10k
- 2:47Data consistency
- 3:34The idea
- 4:17Data flow diagram
- 5:07Parser
- 6:28Admin
- 8:10Stability
- 8:28Offsets
- 9:05Multiprocessing
- 9:48catastrophic crash
- 10:22performance
- 11:00experience
- 11:54Observer
- 12:49Maintenance
- 13:55Loss
- 16:40Questions
related talks
22mIt’s all about the db - a journey to accommodate 100k events per second
33mAnd now to something ELSE: Real Time Data Processing @ billiger.de
40mScaling Python - Bank Edition
28mDownloading a Billion Files in Python
42mBuilding a hosting platform with Python
42mBuilding a hosting platform with Python
01h 02mUsing Python to Power Serverless Applications
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports