Beginning your own data engineering projects
summary
A step‑by‑step recipe for two concrete projects: a Python scraper using BeautifulSoup and Selenium plus pandas and gspread that clicks “show more” 20 times, sleeps between clicks, deduplicates and appends to Google Sheets and collects 1,688 articles over six weeks and runs nightly via Task Scheduler; and an image scraper that saves candidate photos locally and feeds an open‑source GitHub face‑averaging library to produce party “blended” faces when at least ten images exist.
AI-written summary · about
chapters (4)
- 0:00Data engineering projects intro
- 3:46Medium scraper pipeline
- 22:49Blending political faces
- 29:58Key takeaways
related talks
24mimprov BOF How to Scrape Data from any Website
36mThere will be Data: Scraping the Web with Python
32mWeb Scraping Best Practises
22mDistributed Web Scraping in Python
28mData Harvest: Unlocking Insights with Python Web Scraping
35mLittle Data Soup with Spinach (Web Scraping)
19mWeb Scraping for Beginners
02h 51mGathering data from the web using Python