Introducing Cascalog: Functional Data Processing for Hadoop

on YouTube ↗41m

" summary

Cascalog is a Clojure library that builds concise Hadoop jobs by combining functional and logic programming on top of Cascading, so queries use ?vars, generators, predicates and aggregators and run locally or on a cluster. Examples show filtering ages, doubling values and binding results, implicit joins by shared variables (no JOIN keyword), mutual-follow discovers, wordcount via a splitter, and testable queries with Mitch. It lets users use plain Clojure functions as predicates and applies aggregators optimized for map/reduce automatically.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 4:07Hadoop’s MapReduce challenges
3. 9:03What is Cascalog
4. 10:30Core Cascalog queries
5. 14:26Local data processing
6. 16:44Filtering and transformations
7. 22:32Query abstraction
8. 25:28Joins and constraints
9. 28:59Aggregators and groups
10. 35:28Word count example
11. 37:55Summary
12. 38:23Q&A

" related talks

• 46mCascalog: Making Data Processing Fun AgainNathan Marz · Clojure/conj 2011
• 37mCascalog 2- Streaming MapReduce in ClojureSam Ritchie · Clojure/conj 2013
• 36mMoving Faster: Why Intent Media Chose CascalogKurt Schrader · GOTO Chicago 2014
• 01h 12mIntroduction to CascadingPaco Nathan · SpringOne2GX 2012
• 36mApplying Functional Programming Principles to Large Scale Data ProcessingKishore Nallan · Functional Conf 2014
• 33mClojure Dispatches From a National Research LabJennifer Lewis · Clojure/west 2017
• 35mScalable data pipelines with shapeless and catsMarcus Henry, Jr. · Typelevel Summit NYC 2017
• 37mClojure, a Sweetspot for AnalyticsAlex Petrov · EuroClojure 2015