Acidic Jobs: A Layman's Guide to Job Bliss
summary
Argues for making jobs idempotent and transactional by using a job run model that records runs, locks rows, and stores a recovery point so multi-step workflows run stepwise and restart safely; shows transactional staging to enqueue only after commit (so Sidekiq stays), and discovers that Sidekiq batches let steps run in parallel but block workflow progression; offers a pre-1.0 gem called acidic job used in production across multiple projects and demos concrete patterns and minimal code to implement them.
AI-written summary · about
chapters (9)
- 0:00Welcome
- 1:48What defines a job
- 3:31Transactional and idempotent jobs
- 7:29Job uniqueness for idempotency
- 12:23Co-transactional job enqueuing
- 16:34Splitting jobs into steps
- 20:29Parallelizing workflow steps
- 23:40Acidic Job gem
- 24:23Q&A
related talks
30mACIDic Jobs: Scaling a resilient jobs layer
31mLessons Learned Running Sidekiq at Scale
30mBuilding Scalable & Resilient Background Jobs for FinTech
39mHacking Sidekiq for Fun and Profit
27mPrioritization justice: lessons from making background jobs fair at scale
22mSidekiq: Open source, business and the future
11mRunning Jobs at Scale
23mLook Ma, No Background Jobs: A Peek into the Async Future