How to Deploy Machine Learning Inference Code to Production
summary
Code profiling with cProfile finds repeated S3 downloads and 96 per‑sentence model loads, and the pipeline uses torch, SentenceTransformers/E5‑base embeddings and sklearn logistic regression; fixing by pre-download models and caching, offloading writes with FastAPI background task, and batching embeddings so torch and numpy handle matrix ops reduces end‑to‑end latency from ~60s to about 2–3s (≈94–100% speedup). Embedding the whole paragraph is even faster (~2.3s) but hits the model token limit (~512) and changes per‑sentence vs paragraph semantics.
AI-written summary · about
chapters (14)
- 0:00Optimizing ML inference code
- 2:25Fast ML inference motivation
- 5:19System design and infrastructure
- 7:53Production deployment problems
- 10:44Bottlenecks with CProfile
- 14:16Model loading and caching
- 16:24Removing S3 operations
- 18:30Optimizing embedding process
- 23:48Parallel processing in PyTorch
- 27:05Prediction with vectorized ops
- 32:25Memory optimization with NumPy
- 33:08Paragraph-level embedding speed
- 37:37Key takeaways and strategy
- 39:21Requirements and final thoughts
related talks
51mSpeeding up deployment of machine learning models
46mServing machine learning models as an inference API in production
30mProductionizing an unsupervised machine learning model to...
40mBuilding highload ML powered service
29mMachine Learning in Production: What to do after you’ve trained the models?
25mML model from an idea to production with the help...
35mGetting Machine Learning Models Ready For Production Using Python
41mInfrastructure for ML Applications