LLM codegen fails and how to stop 'em
summary
Catalog of actual failures and concrete fixes: models suffer model rot so the system serves fresh markdown pulled by a context service and skill files; agents invent keys and APIs so the pipeline uses model airplanes (thin reference projects) and a CLI-wrapped Claude agent SDK with an LLM gateway; improvisation is limited by breadcrumb the agent sequencing, end‑of‑run interrogation surfaces contradictory tool instructions, and fine‑grained tools (ENV presence check and write only) prevent leaks. This setup supports 5,000+ users a month.
AI-written summary · about
chapters (7)
- 0:00The PostHog Wizard
- 2:12Overcoming model rot
- 4:51Handling bad architecture
- 6:48Limiting improvisation
- 9:31Addressing human error
- 12:07Preventing agent shenanigans
- 14:00Shifting to pros-based design
related talks
38mNobody Pages the LLM: Engineering Rigour for Vibe Coding
41mBuilding LLM Agents – Challenges and Learnings
16mLLMs are not the product
27mThe Lean LLM development playbook
39mLLMs everywhere: The future is now and the past keeps repeating itself
36mLLMs and AI agents demystified
01h 00mFrom LLM Theory to Practical Agentic Implementations
23mLLM Agents in Production: Why Go is the Missing Piece