Lessons Learned Running Sidekiq at Scale
summary
Shows concrete fixes not platitudes: ran into data loss when using Redis replication and built a manual Redis migrator that moves sorted sets in batches and avoids any lost jobs; hit Q‑blocking where 10,000 short jobs stall behind a few minute‑to‑hour tasks and solves it by SLA + component isolated queues with separate ECS services and IAM roles; measures jobs at high resolution (seconds, not minutes) — collect metrics > once per minute and buy Sidekiq Pro/Enterprise for SuperFetch, batches, SSO and operational safety.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:24Separate Sidekiq database
- 5:57Scale Sidekiq queues
- 7:52Isolate queues and jobs
- 11:04Ensure Sidekiq observability
- 16:20Avoid Sidekiq sharp edges
- 22:40Sidekiq alternatives
- 26:30Sidekiq Pro and Enterprise
- 28:18Q&A
related talks
39mHacking Sidekiq for Fun and Profit
22mSidekiq: Open source, business and the future
44mMo' Jobs Mo' Problems: Lessons Learned Scaling to Millions of Jobs an Hour
11mRunning Jobs at Scale
27mPrioritization justice: lessons from making background jobs fair at scale
28mAcidic Jobs: A Layman's Guide to Job Bliss
34mScaling Redis writes with cluster mode
48mOn the Edge of Rails Performance