Celery on AWS ECS - art of background tasks & continuous deployment
summary
Running Celery on AWS ECS Fargate works if workers and tasks are tuned: use stop_timeout=120s and minimum healthy percent 50 to keep capacity during deploys, enable late acks and reject_on_worker_lost=True, target a broker visibility_timeout=15m (aim tasks <15m, empirical 2–3m), set prefetch_multiplier=1 and remap SIGTERM to SIGQUIT for cold shutdown via billiard. Break long work with fan‑out or batching, use Redis locks and retries with exponential backoff and Flower for monitoring; repository examples show this yields continuous zero‑downtime deploys.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:18Async background tasks
- 5:35AWS ECS Fargate - configuration
- 7:06AWS ECS Fargate - interruptions
- 7:48AWS ECS Fargate - continuous deployment
- 9:23Celery on AWS ECS - late task acknowledgement
- 10:15Celery on AWS ECS - visibility timeout
- 12:24Celery on AWS ECS - countdown
- 13:53Celery on AWS ECS - reject task on worker lose
- 16:38Celery on AWS ECS - prefetch multiplier
- 18:00Celery on AWS ECS - signal re-mapping
- 20:07Celery on AWS ECS - example configuration
- 26:43Tasks-batching
- 31:24Working example
related talks
25mMixing reliability with Celery for delicious async tasks
17mBuild a prod-ready distributed task queue system with celery
49mIntroduction to Celery
32mFrom crontab to celery with no regrets
46mA Practical Introduction to Airflow
36mCloudy With a Chance Of Chaos: Verifying the Resiliency Of Cloud-Native Applications
45mPlaying tasks with Django-Celery
26mGet up and running with containers and Amazon ECS