Dream Machine: Scaling to 1m users in 4 days
summary
Luma AI hits a million users in four days by turning a brittle Triton-based stack into a vanilla PyTorch serving stack, manually scaling from 500 to 5,000 H100s in six hours and borrowing another ~4,000 from training to process ~500,000 videos in a launch surge. Operational fixes include a dispatch-limitation to avoid back pressure, a SLO-percent ranking fair scheduler to prevent work starvation, and a model repo with immutable versions and YAML-driven hot swaps for reproducible rollouts.
AI-written summary · about
chapters (10)
- 0:00The initial launch challenges.Luma AI was unprepared for the high traffic, quickly exhausting their initial GPU allocation and facing a large queue of requests.
- 0:57Rapid scaling efforts.They rapidly scaled their GPU capacity from 500 to 5,000 H100 GPUs within six hours, and later added another 4,000 H100 GPUs from their training cluster to keep up with demand.
- 3:10Luma AI's mission.Beyond just video models, Luma AI aims to build general multimodal intelligence that can generate, understand, and operate in the physical world.
- 3:22Their product capabilities.They demonstrate a "modify video" feature where users can upload iPhone videos and transform them with text prompts. They also highlight their public API for integrating this functionality into applications [].
- 6:02Infrastructure re-architecture.They moved from a brittle, tightly coupled container setup using Triton inference server to a custom-built serving stack on vanilla PyTorch, which offers better support for multiple GPUs, nodes, and different chipsets.
- 7:39Challenges and solutions in scaling
- 7:51Back pressure.They implemented a dispatch limitation system to prevent too many CPU workers from queuing jobs in one cluster.
- 8:36Fair scheduling and work starvation.To address issues with different user tiers (API, enterprise, unlimited, light, free) and prevent lower-priority jobs from being starved, they developed an SLO (Service Level Objective) based system that prioritizes jobs based on the percentage of their worst-case waiting time [].
- 13:24Model management.They use a model repository system where each model has immutable versions stored in object storage, including the full Python environment and checkpoints. This allows for reproducible rollbacks and seamless, on-the-fly version switching for workers [].
- 15:13Hiring.Luma AI is actively hiring engineers, researchers, and AI enthusiasts
related talks
37mLive Fireside Chat
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
35mScaling RAG Applications to serve millions of users
06mKeynote: PyTorch Lightning: Powering the GenAI Revolution from Research to the...
26mEfficiently serving LLMs at scale.
12mPreview: ML Serving
23mOperationalizing Large Language Model Training Pipelines