ScrapeGraphAI
summary
Open source ScrapeGraphAI builds scraping pipelines as node graphs and uses LLMs (OpenAI, Bedrock, Llama, Hugging Face) plus a Chromium loader to fetch, convert HTML to markdown and extract structured JSON or Python code. The Smart Scraper flow is fetch → pass → generate; the code generator graph emits ready-to-run Python scripts and Pydantic schemas. Claims: 15K GitHub stars, ~1,200 forks, ~300K pip downloads in eight months. It handles dynamic single-page apps, supports local models, requires a headless Chromium (cloud API coming), and it ignores robots.txt.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:56What is ScrapeGraphAI
- 3:02Smart Scraper workflow
- 4:30Code generator graph
- 5:57Live demo
- 7:01Commercial use
- 8:26Pros and adoption
- 9:58Q&A
related talks
10mFrom Bots to Brains: The Evolution of Web Scraping
01h 26mThinking in Graphs: The Cognitive Foundations of Agentic AI
21mFrom Spaghetti to Structure: Decoding Complex Codebases with Graphs and GenAI
34mEverything is a graph, including LLM Applications (and that’s handy)
28mDelivering AI on Code: Live Demo of source{d}
29mComposing AI Applications as a Graph with Haystack
19mHow to get clean web data for chatbots and LLMs
14mKeynote: Building AI with Open Source & Hugging Face