2025 in LLMs so far, illustrated by Pelicans on Bicycles
summary
A rapid six‑month survey counts about 30 significant model releases and introduces a code‑based test: the Pelican Riding a Bicycle benchmark — prompt models to generate SVG and judge whether a pelican and a bicycle are present. Notable specifics: Llama 3.3 70B runs on a 64GB laptop, Mistral Small 3 is 24B and runs locally, Deepseek drops 685B open weights with a claimed $5.5M training cost, GPT‑4.1 Mini yields cheap quality, a Shot Scraper + LLM CLI pipeline evaluates 500 matchups into an ELO leaderboard, and SnitchBench exposes models that will “rat out” users when given tools.
AI-written summary · about
chapters (13)
- 0:00A review of the last six months in LLMs
- 1:08The "Pelican Riding a Bicycle" Benchmark
- 2:10AWS Nova and Llama 3.3 70B
- 3:30DeepSeek and its impact
- 5:42Mistral Small 3 and the rise of local models
- 6:45Claude 3.7 Sonnet and GPT 4.5
- 8:44Gemini 2.5 Pro, GPT-4o, and Llama 4
- 11:21GPT 4.1, O3, and O4 Mini
- 12:05Claude 4 and other recent releases
- 14:11Amusing and concerning LLM bugs
- 16:58The power of tools and reasoning in AI
- 17:41Prompt injection and the "Lethal Trifecta"
- 18:11The future of the pelican benchmark
related talks
01h 35mSave your API Keys for someone else
22mThe Disruptive Potential of On-Device LLMs
29mRunning Large Language Models on Laptops...
47mLaptop-sized ML for Text, with Open Source
30mLLMs: Beyond the Hype - A Practical Journey to Scale
47mOwn Your LLM: Running Generative and Agentic AI Locally
29mLeveraging open-source LLMs for production
12mKeynote: The Llama Ecosystem: Past, Present and Future