LLM projects: same or different?!
summary
Distills lessons from 10+ LLM projects into concrete production traps: prototypes hide reproducibility and testing gaps, vendor model rollouts (and deletions) break deployed behavior, and embedding indexes force expensive reindexing. Teams rebuild data—e.g., flattening a confusing stock_type column where new used refurbished certified contradicts language—switch runtimes when Python is banned, choose deterministic vs fuzzy testing tradeoffs, and limit runtime or model choices to control cost; recommends reproducibility and treating data/schema fixes as first-class work.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:04LLM vs traditional projects
- 3:53Functional vs non-functional requirements
- 7:46Legal and ethical considerations
- 11:22Controlling LLM outputs
- 13:33Language choice constraints
- 15:42Testing LLM applications
- 21:40Data quality impact
- 29:17Context via agents and training
- 31:03Cost and usage unpredictability
- 35:19Key takeaways
- 35:40Q&A
related talks
15mLLMs for Data-Driven Decision Making in Product Development
30mLLMs: Beyond the Hype - A Practical Journey to Scale
10mRethinking collaboration: LLMs, teams, and cognitive load
33mDesigning LLM Applications for Massive Datasets
29mCustomizing and Evaluating LLMs, an Ops Perspective
24mWill an LLM replace me? A Systematic study of Gen AI's impact on data jobs
32mWhat can we expect of LLMs as software engineers?
19mLLM codegen fails and how to stop 'em