Web scraping in Elixir with Crawly
summary
A new Elixir framework called Crawly builds on Scrapy ideas and aims for production scraping with features like robots.txt by default, request deduplication, item validation and unique IDs, rotating user agents, retries and proxy support. It uses Floki or Meeseeks for parsing, adds browser rendering support for JS sites, and exposes a distributed scheduler plus a dashboard to inspect jobs and scraped items. The claim is seamless scaling on the Erlang VM and easier glue for teams, not just one-off scripts.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 6:59Challenges
- 10:27Toolset
- 13:00Why Elixir
- 15:12Crawly
- 17:18Getting started
- 18:55Defining things
- 19:51What is Crawly
- 20:40Example
- 21:20Crawly configuration
- 22:24Extracting data
- 24:07User interface
- 25:33Summary
- 26:20QA Session
related talks
05mBuilding a search engine: my introduction to functional programming
29mDive into Scrapy
01h 01mWebcrawler
01h 01mLevelling Up Your Web Scraping Game
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
24mimprov BOF How to Scrape Data from any Website
38mScraping LEGO for Fun: A Hacky Dive into Dynamic Data Extraction
08mSEO testing