Effective API schemas testing
summary
A tool called Schemathesis implements property-based testing for OpenAPI v2/v3 and GraphQL by generating Hypothesis strategies from schemas, running requests via CLI or Docker, and checking five built-in assertions (crashes, schema conformance, etc.). It supports stateful sequences using OpenAPI links and Hypothesis state machines. In experiments on 18 services (~1,000 operations) it finds ~20 crashes and ~30 other logic bugs; most schemas (about 97% of schemas) are processable today.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:41Consequences of ineffective API testing
- 1:43How did we get here?
- 2:02Common solutions
- 3:04Formalize testing?
- 3:21Automation
- 3:42Effective solution
- 4:12How to prevent problems?
- 6:20How do we test?
- 6:32Example-based testing
- 7:07Property-based testing
- 8:35API properties
- 9:37API Schema
- 9:53Supported schemas
- 10:12Open API directory
- 12:19Unit tests
- 13:52Open API links
- 14:31Generated API calls
- 15:59Integration tests
- 16:35Why to use Schemathesis?
- 17:43Features
- 18:54Is it effective?
- 19:22In progress results
- 20:29What is next?
- 21:10S SCHEMATHESIS • Schema improvement recommendations
- 21:34Summary
- 22:04References
related talks
19mFunctional Testing of GraphQL APIs: Simple Checks That Catch Real Bugs
30mAPI Evolution With CRDs: Best Practices For Authoring & Fuzz Testing APIs
43mImprove your API and OpenAPI
07m"Using schemas in your API Automation"
45mFuzz Testing Web APIs: Overview of Existing Tools
29mContract Testing in Python for a Data Science API
25mSophisticated Schema Mocking
52mBasics of GraphQL and Testing GraphQL APIs