Why would we ever build a distributed computing platform in node?
summary
Claims that Node can run a distributed compute platform and shows how it actually works: using zeromq for request/reply, round‑robin task distribution and pub/sub, shipping a small open‑source farm module to send/distribute jobs and run generic workers, and using vagrant to reproduce production clusters. A closed launch of ~50–100 users reveals problems at scale (hundreds of machines, GitHub rate limits, mismatched task locality), so the system adopts retries, destroy/recreate instances, metrics via pub/sub, and an operational model that proves workable after iteration.
AI-written summary · about
chapters (33)
- 0:00Introduction
- 0:37Who am I
- 0:48Who are we
- 1:22What does Bit Hound do
- 2:09Time
- 2:35Concurrent
- 2:51Definition
- 3:12Why JavaScript
- 3:45Tiny Hippos
- 4:09Ripple
- 4:40Bait Hound
- 5:39Threading
- 6:47Message Passing
- 7:17Asynchronous IO
- 7:48An inherent cost of IO
- 8:14Asynchronous IO and Node
- 9:14One Thread
- 10:52The short answer
- 11:26ZeroMQ
- 12:41Concurrency patterns
- 13:43Tasks
- 15:19Slaves
- 17:24What we ended up with
- 18:02Farm
- 20:57Launch Day
- 21:26Developer Environment
- 22:34Vagrant
- 23:48Pets or Cattle
- 24:46Fail
- 25:32Rube Goldberg Machine
- 26:09We were a cog in the wheel
- 26:41Why Node
- 27:59Questions
related talks
31mProgramming a Node Cluster Like It's One Really Big Computer
23mBuilding Distributed Systems with Node.js
41mDistributed Systems explained with NodeJS
39mScaling event-driven systems with Node and GraphQL subscriptions
40mDeploying node.js at scale
31mKeynote Panel: Node.js at Massive Scale
22mMulti-tiered Node.js architectures
24mLoad-Balancing Node.js in Production