Scaling generative AI workloads with efficient model choice
summary
Practical model choice workflow that uses Amazon Bedrock to shortlist 3–5 candidates, programmatic and human evaluations (LLM-as-judge at ~98% lower cost), and optional model distillation or fine‑tuning for deployment. Distillation yields up to 500% faster and 75% cheaper student models with <2% accuracy loss on focused tasks. Example: a 7B custom model fine‑tuned on a single A10 GPU for about $30, running ~2x faster than the original foundation model for query generation.
AI-written summary · about
chapters (15)
- 0:00Introduction to scaling AI
- 1:05Amazon Bedrock overview
- 2:52Model selection framework
- 3:55Identifying candidate models
- 8:11Amazon Bedrock marketplace demo
- 9:57Dino Trace use case overview
- 15:27Evaluating model performance
- 17:52Bedrock model evaluation demo
- 21:10Dino Trace evaluation practices
- 24:00Optimizing models for production
- 24:43Model distillation explained
- 26:51Bedrock distillation demo
- 30:24Fine-tuning Dino Trace models
- 38:14Iterative model selection process
- 39:38Closing and resources
related talks
33mThe Right Model for the Right Job: Cutting LLM Costs Without Cutting Quality
39mEvaluate and choose models in Amazon Bedrock (AIM204)
20mBuilding GenAI That Doesn't Cost the Earth
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
35mLLM Customization
39mModel choice in Amazon Bedrock
44mKeynote: Tracking the Pulse of Generative AI
20mFoundation Models for Big Data and Prompt Tuning