How Workday Fine-Tunes LLMs at Scale
summary
Workday builds a fine-tuning platform that trains tenant-specific LoRA adapters on isolated customer data, using DeepSpeed to stitch many single‑GPU instances together and KubeRay autoscaling to spin per‑job clusters via Dagster and Kubernetes. The system serves tens of thousands of tenants (Workday runs for ~65M users) with ~50 production deployments, retrains adapters roughly every two weeks, and accepts a measured 10–20% inference latency hit in exchange for cost savings and per-job clusters isolation.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:14Enterprise LLM challenges
- 7:09Introducing Avato platform
- 9:15Architecture deep dive
- 13:04Multi-tenant security
- 15:33Scaling with Ray and Kubernetes
- 17:43Cost efficiency strategies
- 20:12User experience and SDK
- 21:17Lessons learned hard way
- 24:07Summary and Q&A
related talks
05mInterview
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer
42mStop Treating LLMs Like REST APIs
41mPeer Beneath the Surface of the Earth
22mLN Renganarayana, Architect, ML Platform and Services, Workday San Francisco
30mHow Workload Rightsizing Increases Availability
38mOptimizing LLM Inference for the Rest of Us