How to get clean web data for chatbots and LLMs
summary
Shows concrete wins and failures: using curl impersonate and impit to avoid simple blocks, switching to Playright (browser automation) for client‑rendered pages, and adding stealth plugins or proxies to evade Cloudflare fingerprints. Extracts content with Readability, post‑processes with a countman sketch, converts to markdown, chunks and embeds via LangChain + OpenAI, and demos a website content crawler used in Finn that successfully answers Tesla and BMW manual questions.
AI-written summary · about
chapters (10)
- 0:00Scraping websites for chatbots
- 1:57Fetching raw HTML often fails
- 3:24Avoid getting blocked by sites
- 5:32Dynamic content with JavaScript
- 7:18Advanced bot detection systems
- 9:26Parsing HTML for clean content
- 12:15Scaling web scraping
- 13:39Proxies: data center vs residential
- 15:04Chatbot with scraped data and LLM
- 18:51Q&A
related talks
24mBots & Humans: Maintaining AI agent and automation in an AI-bot web
20mUnderstanding and Mitigating Risks in LLM-Powered Web Apps
01h 01mWebcrawler
53mAI Didn’t Kill the Web, It Moved in!
10mFrom Bots to Brains: The Evolution of Web Scraping
24mimprov BOF How to Scrape Data from any Website
39mHow Chatty Leads Builds AI Agents Businesses Actually Want
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations