Evaluating the alignment of LLMs
summary
City of Amsterdam builds a public assessment platform with tailored Dutch benchmarks and a model table that reports concrete claims: CodeCarbon energy measurements for inference and GPU‑based cost estimates, provider country and license, inclusion tests using nationality/age/gender templates, and a custom honesty benchmark with five categories (latest info, wrong input, expert required, multimodality, missing sources). Small curated sets (~100 prompts) speed evaluation and reduce footprint; current results show no model scoring above ~60% on honesty and closed‑source energy use remains unmeasurable.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:52Amsterdam AI and LLM initiatives
- 4:02Evaluating LLMs for responsible use
- 5:49Key aspects for model evaluation
- 7:15User needs in model selection
- 8:40Building model overview
- 10:09Cost and environmental impact
- 12:37Bias and inclusion in LLMs
- 16:06Factuality and honesty in responses
- 18:35Dutch-specific benchmarks
- 20:20Honesty benchmark for limitations
- 23:12LLM-as-a-judge testing
- 24:56Honesty benchmark results
- 26:28Next steps and benchmarks
- 27:34Political bias and cultural knowledge
- 29:19Status and feedback
related talks
28mTrustworthy LLMs: Vibe checks are not all you need
19mThe Quest for the Greenest LLM
34mBuilding Trust with LLMs
38mMaking LLMs reliable - A practical framework
29mPutting the data science back into LLM evaluation
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
33mPractical Evaluation of LLMs for Code