Scaling Betfair Exchange to support 160k rps using Apache Kafka
summary
Betfair moves its exchange state onto Kafka, replacing a DB-bound monolith so read APIs scale to about 160k rps across 32 nodes and transactional input scales from ~2.5k/s per DB connection toward ~25k/s using Kafka as the input queue. The event model uses in-band snapshot and delta or out-of-band snapshots for large datasets, encodes messages with Protocol Buffers, and achieves long-running reliability and easy rewind/replay for debugging and replay.
AI-written summary · about
chapters (21)
- 0:00Introduction and overview
- 2:09Betfair exchange and growth
- 5:40How betting exchange works
- 8:07Load profile and API demands
- 9:32Legacy architecture limitations
- 12:38Need for new scalable solution
- 13:22Designing Kafka-based event stream
- 14:49In-band snapshot and delta model
- 18:40Out-of-band snapshot strategy
- 21:04Data encoding and consumer libraries
- 22:29Kafka architecture and prototype
- 23:55Kafka cluster setup and reliability
- 26:00Migrating customers to new stack
- 27:46Scaling to 160k requests per second
- 28:29Migration results and benefits
- 29:33Expanding to FanDuel with Kafka
- 31:20Cross-Atlantic catalog distribution
- 33:26Future Kafka use cases
- 35:10Kafka as input queue for bid placement
- 36:32Operational streams and data lake
- 37:34Conclusion and career opportunities
related talks
35mMigrating from Kafka to RabbitMQ at SimpleBet: Why and How
39mUnder the Hood: Engineering at William Hill
38mHow to handle 100k events per second
46mBeyond Betting: How PMU Reimagined Its Data Streaming Service Experience
34mChasing the Stream Processing Utopia
15mPlatform Engineering for FinTech
13mKeynote – Jay Kreps
35mKafka Deployment at Scale