XStream: Stream Processing Platform at Facebook
summary
XStream consolidates Scribe, Puma, Stylus, Laser and Swift into a single managed platform built on Stylus, uses a vectorized evaluation engine from Velox, and adopts interpretation-based execution so one engine binary serves tens of thousands of jobs. Claims: roughly 2x efficiency vs the predecessor; Velox shows 2–10x CPU vs a Java engine; interpretation matches compilation throughput at ~50–100 rows per microbatch with 10–30% less memory and beats compilation by 30–50% throughput when catching up.
AI-written summary · about
chapters (4)
- 0:00Stream processing use cases
- 3:45Brief history of real-time data processing at Facebook
- 8:39Introduction to XStream
- 13:03XStream Design Principles
related talks
39mAdvanced Stream Processing on the Edge
17mComposable Data Management Systems
40mStream Processing and New Approaches: Edge Processing
39mComplex Data Transformations Made Easy
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
33mfStream: a managed stream processing platform at Flipkart
35mIntroducing Prestissimo, Presto's Next-Gen Query Execution Engine
31mBuilding WASM Powered Distributed Stream Platform