Building a Twitter Metadata Scraper: Learnings from Working with a New Twitter API
summary
Builds a service that calls the new Twitter refresh endpoint (beta, whitelist required) whenever Fastly cache purge succeeds, using NSQ message events. Uses Python 3.5, implements a NSQ queue reader, and solves concurrency by wrapping the blocking requests POST in asyncio run_in_executor workers so hundreds of rescrapes run concurrently. Hits initial 429 rate limits, then mitigates them by skipping drafts and pacing requests; outcome: metadata updates happen immediately and reliably in production.
AI-written summary · about
chapters (9)
- 0:00The problem: stale metadata
- 2:07Content management workflow
- 6:33Automated event handling
- 8:24The Twitter challenge
- 10:20Using NSQ for messaging
- 14:03Designing the scraper
- 17:31Implementing the logic
- 20:42Asynchronous requirements
- 23:40Final implementation details
related talks
47mBuilding Scalable Web Apps
24mimprov BOF How to Scrape Data from any Website
28mBoosting Python Web Apps performance with Celery and Concurrency
29mHigh Fidelity Web Crawling in Python
24mDistributed Web Scraping in Python
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
38mUse Apache-Airflow to Build Data Workflows
22mDistributed Web Scraping in Python