Web Scraping with Beautiful Soup
summary
Extracts IMDB pages using Beautiful Soup with requests + lxml and builds a reusable function that takes an IMDB ID and returns a dictionary of metadata. Implements json.loads to pull contentRating from an embedded JSON blob, parses title and summary from HTML, iterates to collect actor and director names, and demonstrates timing differences between parsers. Checks robots.txt, recommends pacing requests to avoid blocking, and delivers a working movie_dict function that successfully scrapes multiple IMDB pages.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:05Robots.txt rules
- 2:37Developer tools for HTML
- 4:01Installing dependencies
- 6:20Setting up project
- 7:45Importing Beautiful Soup
- 10:21LXML vs HTML parser
- 14:02Extracting movie title
- 17:22Finding elements by tag
- 18:49Inspecting elements
- 19:11Extracting movie description
- 20:56Hidden JSON ratings
- 27:55Processing JSON data
- 28:17Extracting actors from JSON
- 33:16Looping actors names
- 34:25Exercise: Scraping directors
- 34:47Reusable scraping functions
- 41:12Movie dictionary function
- 45:05Testing new movie
- 47:30Exercise: Scrape any IMDB movie
- 49:37Beautiful Soup vs Scrapy
- 51:08Why scraping is valuable
- 51:55DataFrame conversion
- 54:06Controlling crawl rate
- 54:52Q&A
related talks
02h 51mGathering data from the web using Python
24mimprov BOF How to Scrape Data from any Website
36mThere will be Data: Scraping the Web with Python
01h 54mPandas Dataframes At The Cinema
28mData Harvest: Unlocking Insights with Python Web Scraping
34mWeb Scraping with Python
19mWeb Scraping for Beginners
01h 07mWeb Scraping with BeautifulSoup & Yelp's API