Testing LLM-Generated Distributed Protocol Code
summary
Evaluates code generated by large language models for distributed protocols—Two-Phase Commit, Ring Election, and Raft—using a simulation that injects network faults (message loss, delay, duplication). Compares LLM behavior on simple protocols and completion tasks versus complex consensus implementations, examines debugging consistency, and studies whether model parameter count predicts performance, arguing for systematic testing and verification.
AI-written summary · about
related talks
38mMaking LLMs reliable - A practical framework
53mSemantics-based program manipulation in the age of LLMs
16mEywa: Automating Model-Based Testing using LLMs
15mInterleaving Large Language Models for Compiler Testing
43mWhat We Talk About When We Talk About Distributed Systems
25mEvaluating An LLM's Ability To Code
35mEvaluating LLM systems: the nuts and bolts
48mThe Verification of a Distributed System