Celery on AWS ECS
summary
Runs Celery on AWS ECS with Fargate, proving zero‑downtime deploys while deploying tens of times per day. Uses Fargate serverless tasks, sets stop_timeout=120s and minimum_healthy_percent=50 to limit interruptions, and configures Celery with visibility_timeout ~15m, late acknowledgements, prefetch_multiplier=1 and reject_on_worker_lost to avoid lost work. Recommends remapping SIGTERM to SIGQUIT via an env var, splits long jobs with fan‑out or batching, enforces idempotency and exponential backoff with jitter; example repo “celery‑ecs‑examples” exists.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 1:08Why background tasks matter
- 3:18Celery basics and AWS ECS intro
- 5:05Fargate benefits for Celery
- 7:37Continuous deployment challenges
- 9:06Critical Celery configurations
- 12:13Countdown and visibility timeout
- 15:51Signal remapping for graceful shutdown
- 18:01Task design for reliability
- 20:09Shortening task duration
- 23:37Resources and Q&A
related talks
35mThe powerful simplicity of Amazon ECS
35mKubernetes on AWS: Zero to hero
27mLeveraging AWS Fargate for running cost-friendly containerized application
40mServerless Compute at the Heart of Your EDA
43mBringing EDA to the Containers World
24mLessons Learned Migrating an On-Prem App to AWS EKS w/ Fargate
27mModernize your applications with serverless, containers, and Amazon EFS
58mFaster, Cheaper, Greener: Pick Three