CI in the Era of AI: From Unit Tests to Stochastic Evals
summary
Zed is a native Rust editor running at 120 fps that moved from exhaustive deterministic CI to stochastic evals for LLM-driven features; tests include a simulated scheduler with 50+ interleavings, programmatic evals run 100–200 times, and deterministic unit tests driven by those failures. Solutions use tree-sitter parsing, fuzzy-matching dynamic programming, indentation normalization, prompt fixes that raise XML-close rates from ~40% to ~95%, and Claude 4 for agentic editing; the system now ships and the eval-driven fixes measurably reduce failures.
AI-written summary · about
chapters (6)
- 0:00Introduction to Zed
- 1:10Deterministic testing roots
- 2:24Embracing stochastic agents
- 5:05Solving streaming edit issues
- 8:10Refining agent behavior
- 12:18Empirical software lessons
related talks
37mAI agents testing How to evaluate the unpredictable
18mCoding Evals: From Code Snippets to Codebases
17mFighting AI with AI
26mBack to the Code Why CTOs Must Get Technical Again to Solve AI’s Real Pro
34mGood or Bad Vibes? Coding with AI in Rust
45mTDD with AI?
42mBeyond the Vibe, How far can we push It?
45mThe AI Revolution in Software Development: From Code Writing to Quality Assurance