Data Pipelining for Real-time ML Models
summary
Optiver builds a custom Python pipeline that ingests about 100 GB per minute from 50 exchanges and produces live ML outputs that match backtests, trading at scale (roughly 3 million trades/day across 100k instruments). The system composes 10,000-edge DAGs of nodes, uses async event loops, multiprocessing, numpy and occasional C wrapping, Redis for raw input, and a hybrid cache strategy—node‑agnostic storage plus kept‑alive in‑memory state—to reach 5000+ predictions per minute and sub-10 second latency.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:32Who am I
- 1:13Why is data important
- 2:08What has changed
- 3:35Why build a pipeline
- 6:46How it works
- 11:05Questions
- 12:58Should you write code
- 15:30How fast
- 17:26Traders
- 21:22Resource Management
- 24:37Production
- 28:32Summary
- 30:52Audience Questions
related talks
24mUnder Pressure: Applying ML in Real Time
38mReal Time ML Pipelines in Multi-Tenant Environments
18mDesigning a scalable ML Ops Pipeline: Insights and best practices
01h 21mBuilding a ML Model Observability pipeline
43mSimplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers
25mContinuous ML Applications in Production
37mAutomated Machine Learning: Building a Pipeline
18mChain-speed inference for Computer Vision Pipelines