Lost in Transduction
summary
Argues transducers cut allocations and run faster than earlier sequence/reducer approaches but are fiendish to implement: reducer functions must use mutation for internal state, must check and propagate reduced values carefully to avoid nested reduced bugs, and must call underlying completion correctly. Shows xforms extensions that make transducers also act as transducing contexts, with a by-key implementation about 30% faster than a naive group-by, and describes a Spark library (Polarkeg) that dynamically ships bytecode to clusters so transducers run on RDDs without a compile step and live coding works.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:07An iteration story
- 6:45Clojure Rationale
- 8:25Transducing contexts
- 10:12chunked vs transducers
- 12:35Informally
- 13:13Clojure transducers
- 13:44The identity transducer
- 14:52Boilerplate init
- 16:42Chain completion
- 21:25The rf is a lie
- 22:37xforms: The Lost Levels
- 23:17xforms transducers
- 24:35x/partition new tricks
- 27:481-pass rollup (advanced)
- 28:10Advanced example
- 29:02Powderkeg
- 30:54Thank you!
related talks
01h 00m“Transducers: from Clojure to C++"
45mTransducers
43mReducers
06mHow to Write a Fast, Reducible Collection
01h 03mUnleash Your Data with Clojure: Using Transducers and Sequences
42mComposable Parallel Processing in Apache Spark and Weld
31mSome Novel Computational Models to Think About
47mKeeping the fun in Apache Spark Datasets and FP