Big Data, Small Code ... #bbuzz

on YouTube ↗34m

" summary

Claims that Java 8 plus Apache Crunch (now shipping the crunch-lambda wrapper in 0.14.0) cuts MapReduce pipeline boilerplate, keeps execution on Hadoop (or Spark), and makes pipelines easy to unit test. Demonstrated concrete tools and patterns: a materialize step that runs a MapReduce stage and streams results back to the driver for Prometheus push‑gateway metrics, and a grouping + streaming top-K algorithm used to build SoundCloud’s stats pipelines. Teams report fewer failures and more tests.

AI-written summary · about

" chapters (16)

1. 0:00Introduction to talk
2. 1:07State of Big Data in 2016
3. 2:34MapReduce is not dead
4. 4:45Apache Crunch overview
5. 7:17Crunch data model
6. 8:01Word count example
7. 9:45Java 8 and Lambda improvements
8. 11:31Crunch Lambda example
9. 12:56Crunch collections explained
10. 15:25Real-world use cases
11. 15:46Monitoring event counts
12. 20:23SoundCloud stats product
13. 26:02Impact at SoundCloud
14. 28:09Future of Crunch and Beam
15. 30:18Summary and resources
16. 32:08Q&A session

" related talks

• 29mBig Data AnalysisSerhiy Masyutin · JavaDay Lviv 2015
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 38mEnabling Big Data and Machine Learning for the Masses...Tim Kral · Scale By The Bay 2018
• 41mBig data as we know it @bol.comDebarshi Basak, Mark Nijenhuis · J-Fall 2015
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015
• 42mFinding Order in Chaos, or MapReduce in the Age of the InternetMonika Nawrot · Confitura 2013
• 23mThe Future of Scala Performance on Big DataCarlos Abad · Scala Days Berlin 2016
• 47mSolving Real & Big (Data) Problems Using HadoopEva Andreasson · GOTO Chicago 2014