Webcrawler
summary
A working web crawler that fetches pages, extracts fields and posts results to a REST API is demonstrated, including a live demo that downloads three daily T‑shirt images and posts them to chat. Implementation details include using cURL for HTTP, HTML parsers with regular-expression fallbacks, PhantomJS/CasperJS to handle JavaScript, and measures like robots.txt respect, connection reuse, timeouts and logging for reliability; tests crawled five domains × ten pages and showed a parallel run (~56 concurrent requests) completed in ~17s versus ~83s single‑threaded.
AI-written summary · about
chapters (11)
- 0:00Webcrawler talk
- 2:18Webcrawler basics
- 9:29Crawl and post images
- 18:48Crawler building mistakes
- 24:05Performance measurement
- 31:59Frameworks vs custom
- 35:29Demo: Parallel vs sequential
- 40:00Crawling JS pages
- 51:53Demo: Extracting JS content
- 57:13Legal and practical limits
- 1:00:39Q&A
related talks
24mimprov BOF How to Scrape Data from any Website
29mHigh Fidelity Web Crawling in Python
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations
01h 01mLevelling Up Your Web Scraping Game
19mHow to get clean web data for chatbots and LLMs
45mGrowing spiders to crawl the web
08mSEO testing
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!