Eywa: Automating Model-Based Testing using LLMs
summary
Eywa uses LLMs to synthesize modular, executable protocol models (converted to code and composed via dependency graphs) and then drives KLEE-style symbolic execution plus differential testing across implementations to validate behavior. It targets DNS, BGP (including confederations and route reflection), and SMTP, producing 110,000+ test cases and finding 33 unique bugs across 16 implementations. The system mitigates LLM hallucinations by composing small modules, generating multiple model variants for diversity, and deriving state graphs to reach target states before testing.
AI-written summary · about
chapters (9)
- 0:00Eywa and model-based testing
- 2:07Manual protocol modeling challenges
- 3:51LLM modeling challenges
- 5:13Protocol modeling with modules
- 7:40LLM hallucinations in tests
- 9:05Stateful protocol testing
- 10:07Bugs in CoreDNS and BGP
- 13:36Eywa results and limitations
- 14:41Future work
related talks
33mExplore LLMs and Multimodal Approaches in Test Automation
15mLLMs and Test Automation: A New Era of Self Healing
47mValidating LLMs – A True Story
21mErly Marsh - a Model-Based Testing Tool
13mTesting LLM-Generated Distributed Protocol Code
38mMaking LLMs reliable - A practical framework
53mSemantics-based program manipulation in the age of LLMs
15mInterleaving Large Language Models for Compiler Testing