Hacking Sidekiq for Fun and Profit
summary
Sidekiq runs on Redis with Celluloid for threading and handles retries and scheduling out of the box; a team pushes unique jobs middleware as a concrete extension to prevent duplicate API calls, and implements a limit_fetch Lua script to atomically acquire work and enforce per-queue limits. Production reaches surprising scale — about 4–5 million jobs first day with zero Redis failures — and experiments with SQS adapters and EC2 spot autoscaling cut costs dramatically but require idempotent jobs and careful monkey‑patching.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:44Hacking Sidekiq For Fun!
- 1:39Decouple from Request / Response Lifecycle
- 2:05Background Jobs in Ruby
- 3:39Threaded
- 4:17Celluloid
- 8:33Scales from a small number of jobs to many
- 9:53Bonus: Testing is simple- instantiate and test worker class.
- 12:41How do we run jobs?
- 15:02Unique Jobs
- 16:36Pushing AND Pulling Jobs
- 18:08Adjusting Running Instances
- 18:55Temporarily pause a queue
- 21:00A smart locking algorithm for jobs in Lua
- 21:10running on your Applications Redis server
- 22:59Replacing the queue
- 30:21Opportunity for custom extensions?
- 30:53Jobs must be idempotent
- 33:49Main Feature: Dead Job Queue For when job fails all retries.
related talks
31mLessons Learned Running Sidekiq at Scale
22mSidekiq: Open source, business and the future
30mBuilding Scalable & Resilient Background Jobs for FinTech
46mFun with Ruby and Redis
28mAcidic Jobs: A Layman's Guide to Job Bliss
27mPrioritization justice: lessons from making background jobs fair at scale
27mGrowing data in a Ruby world
44mMo' Jobs Mo' Problems: Lessons Learned Scaling to Millions of Jobs an Hour