Keynote: LLMs, what are they good for?
summary
Builds a Query Helper that maps natural-language questions to the correct tables in a 58-table schema, using LLMs for semantic search and prompt engineering and validating outputs with Rouge-1 F1 and Jaccard (table intersection over union) against engineer-written dashboards as ground truth. Finds zero-shot OK, relevant few-shot examples markedly improve table/field formatting, OpenAI suffers timeouts so moves to AWS Bedrock with Claude, observes no big quality gaps across models, ships the feature to customers and it proves usable for discovery though not fully trusted for automated chart generation.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 1:34Why LLMs became unavoidable
- 4:46Language models vs human thought
- 9:44LLMs as multi-tools not silver bullets
- 14:43Risks and limitations of LLMs
- 16:48Use case at Pheros: data access
- 20:20Evaluating LLMs for real tasks
- 25:58Choosing the right model
- 28:45Key takeaways and future outlook
related talks
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
47mGive Your LLMs a Left Brain
01h 00mWorkshop: Getting Data Ready for AI - With Unified GraphQL &SQL Endpoints
24mDemocratizing data access through LLM
43mAI Development Lifecycle: Learnings of What Changed with LLMs
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
01h 00mLLMs gone wild
42mKeynote: Designing LLM Native systems