Evaluating Patterns in Gen AI Outputs
summary
Shows why LLMs fail on tiny factual tasks (the strawberry gate counting bug) and why context matters: trained on Common Crawl-like text, transformers use attention not true understanding. Presents concrete limits — productivity claim of 26% from a paper, accuracy drop once codebases exceed ~300 lines — and practical fixes: fine-tuning, temperature/top‑P control, RAG and tool-plugins (calculators, compilers). Offers measured classroom and experiment results (95% less typing on one Rust project) and actionable skills to adopt.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:39Strawberry gate bug
- 15:39LLM vs generative AI product
- 23:43LLM utility for developers
- 38:43Skills for AI-enabled engineers
- 44:22Closing and resources
related talks
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between
01h 59mEmerging Patterns for Coding with Generative AI
37mGenerative AI: Beyond (Just) Hype
39mBack to Basics: Cutting Through the AI-Generated Code Noise
01h 04mWhat are we thinking in the age of AI?
41mGenerative AI: bliss or miss?
15mUsing AI effectively in large Java codebases
49mUsing Generative AI to Strengthen & Accelerate Learning