Boosting Python Web Apps performance with Celery and Concurrency
summary
Uses Celery with Django, a broker (Redis or RabbitMQ), and Postgres to move news scraping off the web app and into background jobs; scheduling uses celery beat and workers run with configurable pools (prefork or threads) and autoscale=10,3 so worker count grows then shrinks. A local project scrapes many sites and exposes an API; measured end-to-end scrape time drops from about 7–10 minutes to ~31 seconds. CLI flags control concurrency and pool mode for IO vs CPU work.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 1:07Web framework purpose and limitations
- 2:31What is Celery and FIFO approach
- 3:54Celery architecture and components
- 5:15Concurrency and parallelism in Celery
- 8:09Celery workers and task distribution
- 9:58CLI commands and pool options
- 12:25Autoscale feature and usage
- 13:50Threads versus processes in Python
- 15:15Why use Celery for web apps
- 16:40Live demo setup and background
- 22:26Live demo results and performance
- 23:32Q&A session begins
