Asynchronous Processing for Fun and Profity
summary
Argues that background work must use tiny, stateless messages (store data in DB, pass IDs) and id-based lookups to avoid stale state, and that jobs must be idempotent and transactional because the processor retries with exponential backoff. Demonstrates a multi-threaded Redis-backed processor called sidekick using Celluloid with ~25 threads per process (four procs → ~100 concurrent workers) that beats process-per-worker systems, saved one user thousands in hosting, but also once knocked over a third‑party API by running 500 jobs at once.
AI-written summary · about
chapters (20)
- 0:00Start
- 2:02Why?
- 3:23How?
- 8:28Tip #1
- 8:39Stateless
- 9:24Avoid State
- 10:03Simple Types
- 10:21Debugging
- 11:09Tip #2
- 11:57Not Idempotent!
- 14:11Tip #3
- 18:11Concurrency . To scale single-threaded, create lots of
- 19:04Threads vs Processes
- 19:53Quick Rant
- 21:08Client
- 21:45Server
- 23:03Versions
- 26:21Pro Future
- 26:40Conclusion
- 31:36Transactional
related talks
47mBackground Processing with
23mDistributed Systems: Your Only Guarantee Is Inconsistency
22mAsynchronous Computing at Scale: Overview and Learning
43mSleeping on the job
53mYou Keep Using That Word: Asynchronous And Interprocess Comms
34mTen-year Journey to 10,000 Production Machines
45mReactive design patterns for microservices on multicore
42mThe Soul of Erlang and Elixir