Concurrent Stream Processing
summary
A production concurrent data-stream engine for database federation processes many streams that are too large for memory by compiling S-expression stream expressions into node-and-pipe graphs executed on a Java fork/join pool. Operators include pmap+, parallel reduce, chunking, re-chunking and join/let semantics; the system swaps to memory‑mapped buffered pipes under low-heap conditions, propagates exceptions to the top, supports cancellation and debugging, and offers alternate compilers (to Clojure seqs or node/pipe trees) used in Revelytics’ shipped product.
AI-written summary · about
chapters (15)
- 0:00Start
- 16:14Direction of Data Flow
- 19:42Processor
- 21:25Cancellations / Timeouts
- 22:24Compiling Stream Expressions
- 23:11Stream Operators Operators for taking parts of streams.
- 23:21Stream "let" Operators
- 24:52Stream "Chunk" Operators Chunk operators allow operations on chunks
- 29:29Inside the Stream Expression Compiler
- 30:28Compiler Implemented via eval
- 34:09Code Generation
- 35:42Distributed Stream Processing
- 36:04Related Clojure Links
- 39:09Efficient Use of Worker Threads
- 40:27Exception Handling
related talks
44mThinking in Streams
49mConcurrency and Streaming in the Age of Loom
48mParallel Streams, CompletableFuture, and All That: Concurrency in Java 8
48mStreamly: Declarative Concurrency And Dataflow Programming In Haskell
41mParallel Programming, Fork Join, and Reducers
39mLet's talk concurrency
56mFlavors of Concurrency in Java
49mBEAM Concurrency in Action