Functional Big Data
summary
Functional programming maps directly onto big data: MapReduce is the functional pattern, Scalding offers a concise Scala DSL that compiles to MapReduce, and Apache Spark exposes RDD (Resilient Distributed Dataset) APIs that look like regular collections but run partitioned across a cluster and cache in memory for orders of magnitude speedups. Examples include distributed word count, processing JSON event archives on S3, submitting an application jar to a cluster, and Spark Streaming with batch interval 1s and simple idle detection 15s alerts.
AI-written summary · about
chapters (7)
- 0:00Introduction to functional Big Data
- 2:29Functional programming overview
- 10:41Apache Hadoop explained
- 19:13Apache Spark introduction
- 28:20Real-world Spark applications
- 31:07Spark Streaming overview
- 41:55Summary and conclusion
related talks
40mFunctional Programming for Machine Learning and Data Pipelines
01h 01mTransforming Data using the Functional API
01h 50mObject-Functional Patterns in Scala
49mMining Functional Patterns
47mKeeping the fun in Apache Spark Datasets and FP
01h 00mWhat The Functional Programming
44mBenefits of Functional Programming Beyond map/filter/reduce
57mThe Fun of Functional Programming Panel