Big Data with Small Computers: Building a Hadoop Cluster with Raspberry Pis
summary
A home Hadoop cluster built from 3 Raspberry Pis runs HDFS, YARN and Apache Spark and successfully executes small Spark jobs (hello‑world reading a file from HDFS), but setup reveals concrete pitfalls: use a real switch instead of Wi‑Fi, enable SSH on each Pi, format SD cards correctly (>64GB issues), and pin Java to Java 8 for compatibility. The stack used is Hadoop, YARN, HDFS, Spark; Hadoop 3 changes ports and names, logs are very verbose, and the cluster works intermittently.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:54Why do this
- 1:56Raspberry Pi example
- 2:23Other examples
- 3:34What is Hadoop
- 5:50Hadoop Ecosystem
- 7:45HDFS
- 8:35Replication
- 9:30Resource Manager
- 10:20Compatibility
- 13:42Troubleshooting
- 14:16Screenshots
- 14:52Networking
- 16:48User Interface
- 18:56Jet
- 19:23Spark
- 20:31Resources
related talks
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
43mKeynote: Tiny Data Collectors, Vastly (...) & the Land of Tiny Challenges
41mHands on Introduction to Distributed Systems Concepts with Akka Clustering
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
59mDistributed applications using Hazelcast
09mDebugging Hadoop
50mBig Data on Smallish Machines