Web Scraping Best Practises
summary
Python tooling lineup is concrete: Scrapy for crawling with sessions retries and autothrottle, Frontera for frontier state at scale, and Portia or webstruct (trained with WebAnnotator and CRFs) for extraction. Real outcomes include crawling hundreds of millions of pages and discovering tens of billions of URLs, a Saatchi Art project scraping 11,000 websites that uses image face recognition classification to separate artist photos from artworks and that is working pretty well with measured accuracy and iterative improvements.
AI-written summary · about
chapters (21)
- 0:00Start
- 0:22Web Scraping Best Practices
- 1:10Why Would You Want To Scrape
- 3:05Example of Web Scraping
- 4:35Python Requests Library
- 4:54Retrying Requests That Fail
- 5:22Using a Session
- 9:37Incremental Crawling
- 12:09Extraction
- 12:37Regular Expressions
- 13:19Html Parsers
- 13:38Html Parser
- 13:52Examples of Html Parsers
- 15:14Beautiful Soup
- 18:30Annotation Web Pages
- 24:09Selenium
- 24:22Web Inspector
- 24:32Proxy Management
- 25:44Ethics of Web Scraping
- 26:51Logging into Websites
- 31:43Dns Errors
