Supercharging ML Pipelines with Go
summary
A Go-based service initializes shared memory shims and packs four tensor types (int64, float32, int8) into batched layouts with checksums, handing them to a Python ranking model via the Trident inference server. The pipeline writes ~40 million records (~4 GB) to memory-mapped files in about 50 seconds, enabling end-to-end cohort generation in roughly 10–15 minutes instead of hours; implementation includes worker pools, channels, and simple mmap copy/cleanup routines.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:32ML platform overview
- 1:51Individual recommendations
- 2:25Ranking challenge
- 3:24Designing the solution
- 4:41Implementing shared memory
- 6:48Infrastructure setup
- 7:15Data contract definition
- 7:59Batching for performance
- 8:54Writing data to SHM
- 9:40System performance
- 10:21End-to-end workflow
- 12:33Conclusion and results
related talks
40mBuilding highload ML powered service
16mRevisiting Pipeline Parallelism for LLM Serving
22mOne Billion Row Challenge in Golang
23mHeterogeneous Pipelines as the Unlock for ML at Scale
36mAccelerating High-Performance Machine Learning at Scale i...
47mWill your infrastructure support your ML initiatives?
24mTaming Data Skew in Production ML Pipelines
26mEfficiently serving LLMs at scale.