Python tools for webscraping
summary
Shows concrete tools and a working Scrapy project that actually crawls the PyData Madrid site, defines items for title speaker time and description, and pipes results to JSON or SQLite; measures and claims Scrapy uses Twisted and lxml so it outperforms simple parsers when making many requests. Demos include requests + BeautifulSoup, PyQuery, Selenium + PhantomJS for JS pages and mentions PDFMiner for PDFs. Example deploys to Scrapy Cloud and the pipeline output works.
AI-written summary · about
chapters (18)
- 0:00Python web scraping tools
- 2:22Web scraping techniques
- 4:12Screen scraping with Selenium
- 8:34PDF extraction
- 9:18Key Python libraries
- 13:27Effective scraping steps
- 15:39Beautiful Soup for HTML
- 19:23Scraping with requests and lxml
- 22:42PyQuery for extraction
- 24:34Scrapy framework
- 28:32Scrapy architecture
- 31:22Scrapy project setup
- 33:14Building a Scrapy spider
- 36:56Scrapy pipelines
- 38:46Running Scrapy spiders
- 40:36Deploying to Scrapy Cloud
- 42:28Further reading
- 43:13Q&A on PDF scraping
related talks
02h 51mGathering data from the web using Python
29mDive into Scrapy
32mWeb Scraping Best Practises
28mData Harvest: Unlocking Insights with Python Web Scraping
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
35mLittle Data Soup with Spinach (Web Scraping)
01h 01mLevelling Up Your Web Scraping Game
24mimprov BOF How to Scrape Data from any Website