Introduction to Celery
summary
Covers hands‑on Celery setup: a real app uses a task to generate thumbnails and another to build PDFs, tasks queued via a broker and executed by workers; recommends RabbitMQ for durability over Redis because Redis can drop or duplicate tasks, shows periodic scheduling (example: 5 minute reminders), default consumer prefetch is 4, concurrency is usually “n+1” worker processes, warns against pickle and passing objects (use IDs), says you probably need no result backend until you really do and suggests multiprocessing over green threads.
AI-written summary · about
chapters (4)
- 0:00Intro
- 1:59Major Features Allows out of band processing, started periodically or trigged by web requests Communicates with a message broker and distributes work Can act as a cron job replacement Can help ensure requests are timely Reasonably good at distributed computing and high availability
- 16:38The Role of the Message Broker and Choosing One
- 35:10Parallel Safety Write tasks to be parallel safe. You have no idea when tasks will run Plan for this. Trying to disallow certain tasks from running in parallelis focusing on the wrong problem. Use database transactions where needed Don't pass objects as task parameters.
related talks
45mPlaying tasks with Django-Celery
40mHow to Schedule Tasks with Celery and Django
42mCelery on AWS ECS - art of background tasks & continuous deployment
32mFrom crontab to celery with no regrets
45mPlaying tasks with Django-Celery
28mBoosting Python Web Apps performance with Celery and Concurrency
19mOptimizing Data Retrieval with Python Celery
30mBuilding data flows with Celery and SQLAlchemy