Becoming a Spider
summary
A working web crawler that fetches pages using axios and parses them with cheerio, follows links into a queue while throttling requests and avoiding loops, and delegates work to serverless functions that publish jobs via AWS SNS/SQS. Extracted content and computed keyword-density scores are stored in S3 index storage, with the extractor applying selectors; the full pipeline is demonstrated end-to-end as it crawls and indexes pages.
AI-written summary · about
chapters (4)
- 0:00Start
- 4:04Anatomy of a refinery
- 10:46Refinery stage 1: The Trigger
- 29:46Refinery stage 4: The Indexer
related talks
08mSEO testing
45mThe Art of Web Scraping. Trees, Algorithms and JS Magic
06mApify SDK - The scalable web crawling and scraping library for JavaScript
57mChrome Extension for Page Scraping & Analysis
43mBack to Basics Let's Code some web stuff
28mJavaScript SEO (The "Right" Way)
24mMaking It Better Without Making It Over
46mMass Capture of User Experience with JS