Data-flow Processing for Node.JS with Straw
summary
Straw claims to be a lightweight data‑flow framework for Node.js that already runs on npm and is used in production to process millions of messages a day. It uses Redis lists with BRPOP for queues and built‑in load balancing, runs each node as forked child processes to use multiple cores, and exposes taps so an Express + socket.io front end can stream e g a Twitter firehose map, top‑20 hashtags and language counts. Real deployments (electricity futures feed) handle spikes and auto‑restart crashed nodes.
AI-written summary · about
chapters (39)
- 0:00Intro
- 0:42Computer
- 3:09IBM
- 3:33Twitter
- 3:53Recap
- 4:08ENIAC
- 5:02MAX MSP
- 5:34Why MSP
- 6:18Straw
- 6:33Hello World
- 7:04Topology
- 7:52Structure
- 8:28CountOut
- 8:40Print
- 9:08Start with Straw
- 9:31Processing incoming messages
- 9:52Twitter Firehose
- 12:24Taps
- 13:16Socket
- 13:31Haystack
- 13:57Github repo
- 14:19Redis
- 14:58Throw
- 16:17Advantages of NodeRunner
- 17:43Creating a runner
- 18:04Running a child process
- 18:34Communication with the process
- 18:58Command line arguments
- 19:24ReadUs
- 19:49Redis Lists
- 21:53Event Handler
- 22:09Redis Monitor
- 22:54Real World Example
- 24:27Screenshots
- 24:45Pipe Monitor
- 25:41Stats Deep
- 26:17Charts Deep
- 27:05Summary
- 27:19Wrap up
related talks
23mDecomposition of the Main Thread in Node.js to increase throughput
31mBelieve in Streams
34mYou Don't Know Node.js
26mTechniques and Tools for Taming Tangled Twisted Trains of Thought
31mClojure at 4,000 msg/s-What We Learned, Loved, and Loathed
22mEvery Node.js app should have real-time collab features
35mStreams: The data structure we need
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets