Selecting the Best LLMs with Model Cockpit
summary
Freshworks ships a persona-based Model Cockpit (UI + API) that centralizes model selection, licensing, benchmarking, staging and promotion into production, with a locked prompt library that logs requests for retraining. Demo shows CodeLlama 34B beating Llama 7B for code; a fine-tuned FM summarization model on Mistral produces concise bullets. Governance blocks abuse and enforces fairness. Ops use MLflow/Kubeflow, model registry, fractional GPU and red pill/blue pill economics; self-host runs ~$20k/month, break-even versus managed services appears near 200k calls.
AI-written summary · about
chapters (7)
- 0:00Introduction to Model Cockpit
- 3:14What is Model Cockpit
- 7:02Red pill vs blue pill analogy
- 9:27Economics of self-hosted vs managed LLMs
- 12:23Model lifecycle with Model Cockpit
- 15:54Demo of Model Cockpit interface
- 22:18Managing self-hosted LLMs
related talks
33mThe Right Model for the Right Job: Cutting LLM Costs Without Cutting Quality
59mWorkshop: Choosing the Best Open Source LLM for Your Application
28mNavigating LLM deployment: Tips, tricks, and techniques
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
29mLeveraging open-source LLMs for production
40mKeynote
31mDeploying Large Language Models with Ease
50mPimp My Llama - How to Customize LLMs for Your Business Case