Powderkeg: teaching Clojure to Spark
summary
Powder Keg runs Clojure transducers directly on Spark RDDs, sending dynamically compiled classes and a snapshot/log of namespaces and vars to workers so REPL-driven code works on a real cluster without AOT. It uses a small Java agent to extract bytecode, an instrumentation hook to stream var updates, and xforms-style by-key transducers to do single-pass aggregations that avoid tuple allocations. Demo cluster runs succeed though mutable pre/post-shuffle logic needs care; streams and DataFrame support are unfinished.
AI-written summary · about
chapters (5)
related talks
32mData-Shuffler: Bringing Order to The Galaxy (of Data Process)
31mSpark challenges
37mClojure Remixed: Pyroclast Data Processing
29mDynamic Large Scale Spark on Kubernetes
20mPigPen: Map-Reduce for Clojure
30mBeyond Experimental: Spark on Kubernetes
21mAvoiding Spark Pitfalls at Scale
42mBuilding Robust Streaming Data Pipelines with Apache Spark