How to train a fast LLM for coding tasks
summary
JetBrains trains a purpose-built in‑IDE code completer called Mellum, a ~4B parameter model fine-tuned for fill‑in‑the‑middle code completion and context awareness using a Kotlin-based “code engine” to gather project files and context. Pretraining uses ~4.5T tokens and an 8k window, then supervised finetune with realistic split heuristics, license filtering (remove non‑permissive repos), alignment via preference scoring from larger models, quantization/distillation and SFT. Trained on 64 GPUs ×4 weeks (or 128 ×2), it’s cheaper, faster and outperforms models about twice its size on multi‑line completion metrics.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 2:29Why build custom SLMs
- 6:27Training from scratch vs alternatives
- 8:56Code completion fundamentals
- 12:12Pre-training process details
- 14:43Supervised fine-tuning steps
- 18:13Context awareness challenges
- 21:29Alignment and human-like generation
- 27:16Results and model lineup
- 29:03Q&A session
related talks
29mBeyond Autocomplete: Local AI Code Completion Demystified
01h 05mThe Next Coding Horizon - Proactive LLMs for Enhanced Productivity
27mTeaching a Machine to Code
45mEnhancing LLM Powered Development with Clojure's REPL
10mWhat goes into the making of LLMs
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
29mLocal LLM fine-tuning: a practical example
25mEvaluating An LLM's Ability To Code