"Context Aware Content Discovery: The Natural Evolution"
summary
Collects and normalizes public OpenAPI/Swagger specs (BigQuery, apis.guru, SwaggerHub, internet scan) into a 67.5k-spec corpus and ~1M unique routes, then generates contextual requests (methods, path params, headers) from those specs. Uses masscan, zgrab2, custom parser and heuristics to suppress wildcards and false positives, and runs distributed scans at ~26–30k requests/sec across hosts. Releases the dataset and the tool kite runner plus converters; example finds include an integer-path file disclosure and a POST returning 500 revealing a valid endpoint.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:43The Importance of Content Discovery
- 1:59Evolution of Content Discovery
- 3:57Why Context Matters
- 8:16Context Aware Content Discovery
- 8:53Our Approach
- 12:39Scanning the Internet
- 14:45Scanning Process
- 15:31Custom Tooling
- 18:01Conceptualising a schema for the data
- 18:54What does this output look like?
- 19:07Writing the tool to parse the data
- 20:11Generating the final output
- 23:43Handling Wildcards
- 27:46Scaling Out Scanning
- 31:06the goodie bag
- 36:52With Kiterunner
- 37:46Traditional Content Discovery Tools
related talks
53mUncovering Swagger/OpenAPI
24mAPI Discovery: Turning Hidden APIs into Strategic Assets
35mOpensource API Tools to agile your apifirst strategy.
21mAPI Discovery done right: Engaging the right people with your APIs.
19mAutomate your API to perfection
23mCross-platform Catalogs for Unified Governance and Developer Experience
19mGenerating & Using AsyncAPI Specs from Real-World API Data
27mGetting developers hooked on your API