The Next Frontier for AI-Generated Kernels: Correctness
summary
KernelBench’s fixed test suites let buggy kernels pass tests and even reject correct implementations; mutation testing finds racey softmaxes and trivial zeroing that still match references. The project uses a two‑phase agent pipeline (generate+test, then verify) with Claude 4.x to produce Kuiper verified implementations that guarantee memory safety, data‑race freedom and functional correctness, extract to CUDA and compile with NVCC. Results: 100 KernelBench Level 1 tasks fully verified, written almost entirely by LLM agents.
AI-written summary · about
chapters (17)
- 0:00AI kernel correctness intro
- 1:39GPU kernel challenges
- 3:02Testing limits for kernels
- 6:55AI-verified kernels with proofs
- 9:22Two-phase kernel verification
- 13:56KernelBench verification results
- 16:25Why testing fails kernels
- 17:33Q&A: Teaching Kyber language
- 18:57Q&A: Verified kernel performance
- 19:40Q&A: Trust in specs and errors
- 21:46Q&A: Mega kernel scalability
- 22:49Q&A: Handling NaN bugs
- 24:56Q&A: Human review of specs
- 25:38Q&A: Spec correctness and traces
- 26:21Q&A: Formalizing inline PTX
- 27:03Q&A: Polymorphism and float types
- 28:28Closing remarks
related talks
19mAI Kernel Generation: What's working, what's not, what's next
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
20mUnlocking Performance: Harnessing LLMs To Streamline GPU Kernel Development in...
44mConfidently Wrong: When AI Cannot Catch Its Own Bugs
18mYour Coding Agent Should Do AI System Engineering
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
46mAI for Kernel Engineers
32mCan We Trust AI-Generated Code?