Becoming AI engineers
summary
Claim: a production-ready automated incident investigation and resolution system is built over 18 months that routinely runs ~109 LLM calls per investigation. Tools actually used: a prompt abstraction layer with multi-provider support and structured-output typing, eval suites that use LLM graders, AI traces storing inputs/outputs and model params, scorecards and back tests run against 29 historic incidents. Result: a deploy-and-test loop that yields a 15% improvement in recall on code-change detection.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:55What is AI engineering
- 9:22eval tests
- 12:43our app
- 14:14trace view
- 17:11scorecards
- 19:59making changes
- 22:50celebrating
- 24:09outro
related talks
30mAI and Incident Management: How to Reduce Manual Work and Speed up the Incident...
36mRisks and Opportunities of AI in Incident Management
05mAI-Driven Incident Resolution – Hype or Reality
26mAI & SRE: Embracing the Ultimate Frontier of Automation
43mAI Development Lifecycle: Learnings of What Changed with LLMs
19mFrom Alerts to Action: AI-Powered Incident Response Systems
30mAI Agents for Incident Management
22mThe AI Productivity Paradox in Engineering