Introduction to Cascading

on YouTube ↗01h 12m

" summary

Cascading is a Java application framework that compiles entire Hadoop workflows into one JAR file, with integrations for HBase, Avro, Thrift, JDBC and deployments at hundreds of sites (Twitter, Etsy, Williams‑Sonoma). Examples move from a 10‑line file copy to a full tf‑idf pipeline and a Twitter‑firehose social recommender that parallelizes a naive cross‑product and writes results into Redis/Neo4j. Features include stream assertions, unit tests, checkpoints for long runs (one customer uses a 1000-node cluster with 250 job steps taking a week) and AWS EMR deployment.

AI-written summary · about

" chapters (41)

1. 0:00Intro
2. 0:38Enterprise Apps for Big Data with Cascading
3. 2:09Cascading API: purpose
4. 2:36Cascading API: a few facts
5. 5:26Cascading API: a few quotes
6. 6:20data+code "political spectrum" "Notes from the Mystery Machine Bus" by Steve Yegge, Google goo.gl/SeRza
7. 8:26Cascading API: adoption
8. 8:49enterprise data workflows
9. 10:16data workflows: team
10. 11:27data workflows: layers
11. 13:11data workflows: SQL VS. JVM
12. 16:49inflection point: consequences
13. 18:26primary sources Amazon
14. 18:53data innovation: circa 1996
15. 19:43data innovation: circa 2013
16. 25:08statistical thinking
17. 28:21MapReduce architecture
18. 29:26MapReduce: how it works
19. 31:29a brief history...
20. 33:38CAP theorem purpose: theoretical limits for data access patterns
21. 36:37parallel computation
22. 38:25core values
23. 39:13most valuable skills
24. 41:21the science in data science?
25. 44:03team process = needs
26. 44:44team composition = roles
27. 44:56matrix: example team
28. 45:25typical hand-offs
29. 45:42use case: marketing funnel
30. 46:33use case: ecommerce fraud
31. 47:48use case: customer segmentation
32. 48:01use case: monetizing content
33. 48:56reference
34. 50:18copy
35. 51:03same JAR, any scale...
36. 52:11word count
37. 52:41Cascading / Java
38. 53:26Scalding / Scala
39. 53:52Cascalog / Clojure
40. 54:55Hive
41. 59:41tf-idf + tdd

" related talks

• 36mMoving Faster: Why Intent Media Chose CascalogKurt Schrader · GOTO Chicago 2014
• 41mIntroducing Cascalog: Functional Data Processing for HadoopStefan Hübner · EuroClojure 2012
• 46mCascalog: Making Data Processing Fun AgainNathan Marz · Clojure/conj 2011
• 37mCascalog 2- Streaming MapReduce in ClojureSam Ritchie · Clojure/conj 2013
• 42mBuilding Robust Streaming Data Pipelines with Apache SparkZak Hassan · Open Source Summit North America 2017
• 59mSimplifying Big Data Management with Apache FalconSrikanth Sundarrajan · Great Indian Developer Summit 2015
• 59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & AkkaHelena Edelson · Philly ETE 2015
• 46mScalding Big Data: Thorns and RosesBoris Trofimov · JavaDay Lviv 2015