Ruby Queue
summary
A command‑line tool called rq sets up instant Linux clusters using a shared nfs hierarchy and an embedded sqlite queue, with custom gems for POSIX locks and a slave process manager. The system runs on 30–50 nodes, handles gigabyte bands and workflows that generate hundreds of terabytes of intermediates, uses nonblocking sawtooth locking and lease refreshes to avoid nfs fairness and dot.nfs issues, boots from cron for zero‑admin daemons, and proves robust through hardware failures.
AI-written summary · about
chapters (33)
- 0:00Intro
- 1:31Who am I
- 2:17Project Management
- 3:22GDC
- 4:28Data
- 5:16Ruby Queue
- 7:27Learning
- 8:00What is RQ
- 8:15What is Linux Cluster
- 9:17Change Image
- 10:45Why Ruby Queue
- 15:17Architecture
- 19:44Hotspot Development
- 21:30LockFile
- 23:11LockD
- 24:48Mass Key Art
- 26:39Lock Policy
- 27:15SQL Lite
- 29:36Demo
- 31:46Start a Feeder
- 32:34NFS
- 33:35Database API
- 35:04Slave Process
- 36:27Grid
- 37:14Feeders
- 37:56Queue cron
- 38:47Queue features
- 40:21Status statistics
- 42:10Standard Error
- 42:39Installation
- 44:01Future Plans
- 45:07Lessons Learned
- 49:24Questions
related talks
43mRails is a Follower
26mRuby off the Rails: Building a distributed system in Ruby
46mRuby in the Clouds
26mRocky Mountain Ruby 2023 - Lightning Talks
27mReady: 12x Faster Ruby CLIs Through Spectacular Overengineering
39mPractical Unix for Ruby & Rails
01h 03mRuby at Scale
30mGrow Your Unix Beard Using Ruby