PigPen: Map-Reduce for Clojure
summary
Map-Reduce for Clojure that lets you write plain Clojure, ship closures and immutable params to the cluster, and run the same code locally for quick feedback. It generates Apache Pig scripts or executes locally via an RX-backed evaluator, supports distributed idiomatic Clojure and iterative REPL-driven development, and has full unit/generative test support. PigPen runs on thousands of machines, composes branching/looping pipelines, handles queries that compile into hundreds of Hadoop jobs, and is used in production at Netflix.
AI-written summary · about
chapters (8)
- 0:00PigPen for Clojure
- 3:12Distributed Clojure
- 7:56Word count example
- 10:01Expression building and optimization
- 12:50Joins and co-groups
- 15:09Parameterizing functions
- 17:16Visualizing workflows
- 19:00Submitting jobs and future plans
related talks
35mPowderkeg: teaching Clojure to Spark
34mClojure by neophyte
41mParallel Programming, Fork Join, and Reducers
39mPersonalized Medicine, Powered by Clojure
37mClojure Remixed: Pyroclast Data Processing
37mDon't Shed a Tier for Me, Clojure
43mDoing Data Science with Clojure
37mCascalog 2- Streaming MapReduce in Clojure