From Big Data to Big Information
summary
Hadoop remains the low‑cost, batch engine for massive scans (used at Facebook on ~600 PB and chosen to replace a $1M data warehouse to keep years of history on a 160‑node cluster), but MapReduce forces disk writes between steps and a verbose Java API. Spark caches data in memory and can run 30–100x faster for iterative jobs. Hive and Impala provide SQL on Hadoop for analysts. Storm handles real‑time stream processing, HBase gives record access on HDFS, and ElasticSearch/Solr or Titan/Neo4j solve search and graph needs.
AI-written summary · about
chapters (16)
- 0:00Introduction to big data
- 0:50Defining the big data era
- 1:34Evolving data architectures
- 2:39Data-driven algorithms
- 5:07Hadoop and ecosystem tools
- 11:10Understanding MapReduce
- 15:56Alternative tools and APIs
- 21:28Hadoop vs databases
- 23:23Hadoop as a warehouse
- 26:44SQL on Hadoop
- 30:07Search capabilities
- 30:40Real-time processing
- 32:56Machine learning use cases
- 35:37Graph data analysis
- 39:43Future trends and summary
- 43:48Q&A session
related talks
51mBIG DATA STORAGE & PROCESSING: Introduction to Hadoop and Its Ecosystem
47mWhat's Ahead for Big Data?
15mOptimizing Query Performance in Big Data Systems
39mBig Data Systems Performance: The Little Shop of Horrors
59mHadoop Essentials for the SQL Server Professional
08mHadoop and Big Data Hype
50mFast Analytics On Big Data
07mIn-Hadoop Analytics: Bringing analytics to big data