Beyond Betting: How PMU Reimagined Its Data Streaming Service Experience
summary
PMU migrated its horse-racing data platform to AWS and built a production streaming pipeline that processes ~70,000 races/year, ~4.7M bets/day and bursts up to ~8,900 bets/sec with sub-500ms delivery goals. The solution uses Kafka on AWS MSK for real-time feeds, Amazon Glue with Spark writing Parquet for batch quality, Lambdas plus Quarkus and Spring Boot for ingestion, DynamoDB for hot reads, API Gateway for per-contract filtering, and an Avro schema registry to manage versioned contracts for 110+ partners.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:04Project context and stakes
- 8:29Data architecture
- 13:07Data processing pipeline
- 17:23Reference data management
- 23:43Real-time streaming
- 30:28API contracts management
- 36:03Future perspectives
- 45:16Q&A
related talks
39mScaling Betfair Exchange to support 160k rps using Apache Kafka
44mBig Data with AWS: our lessons learned using managed services
39mUnder the Hood: Engineering at William Hill
35mMigrating from Kafka to RabbitMQ at SimpleBet: Why and How
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
34mChasing the Stream Processing Utopia
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns