Deploying Your Own Language Models: A Use Case Demo with Open Source Tools
Talkat Open Source Summit Europe 2023 · on Wednesday, 2023-09-20· at 15:55· Room 0D-2-0D-3 (Floor 0), Abandoibarra Etorb., 4, 48011 Bilbao, Bizkaia, Spain·
summary
A domain QA system for ROSA docs is built and demonstrated end-to-end: docs are encoded into 384‑dim embeddings, indexed in Qdrant vector DB, top‑k candidates are retrieved and fed to an LLM for answer synthesis. Initial pipeline uses OpenAI GPT‑3.5 and shows hallucinated links (2 of 3 wrong); an open source experiment fine‑tunes Flan‑T5 783M on one T4 16GB GPU and is served with Ray Serve deployment. Evaluation uses NDCG and BLEU plus human feedback (Argilla) and highlights the tradeoff between model size, GPU cost, and answer fidelity.
AI-written summary · about
chapters (18)
- 0:00Deploying language models
- 1:11Introduction
- 2:36LLMs and transformers overview
- 5:29Documentation search QA app
- 6:54Live QA demo
- 9:03Embeddings and vector search
- 13:18QA app architecture
- 14:43Vector databases for search
- 16:12Answer evaluation with prompts
- 17:56Adapting open LLMs
- 18:37Pre-training and fine-tuning
- 20:47Selecting open models
- 22:51Deploying Flan-T5 on Kubernetes
- 29:42Model performance metrics
- 30:47Human feedback importance
- 32:56Using feedback in apps
- 33:18Key steps to build app
- 37:24Q&A
related talks
40mDeploying Your Own Language Models: A Use Case Demo with Open Source...
43mLarge Language Models for Devs: from zero to your first LLM application
51mLarge Language Models for Real-World Applications - A Gentle Intro
40mKeynote: Open-source Multimodal AI in the Wild
31mDeploying Large Language Models with Ease
50mProduction-Ready GenAI with Open Models for Java Teams
34mBuilding Trust with LLMs
01h 00mUnleashGPT: Mastering the Implementation of Large Language Models