Deep Dive into the Big Data Landscape
summary
Concrete pipelines and outcomes: structured sales CSVs load with Scoop -> Avro -> Hive -> Impala to produce interactive top-sellers; web clickstreams ingest with Flume -> SolrCloud with Morphlines to index in real time and detect issues (e.g., traffic anomalies that explain sales drops); Spark examples compress an inverted-index job to a few dozen lines as an RDD pipeline and enable 100x speedup via in-memory caching vs MapReduce, plus Spark Streaming micro-batches and Spark SQL for ad hoc queries.
AI-written summary · about
chapters (25)
- 0:00Start
- 1:20Today's Big Data Landscape Journey
- 2:49Data Re-Thinking Drivers
- 8:48Existing Technology Failing?
- 10:37Technology Evolution
- 14:45Hadoop Distribution Vendor Evolution
- 16:56Snapshot of the Data Management Landscape
- 26:35New Category: The Enterprise Data Hub (EDH)
- 27:50Hadoop et al Enabling an EDH
- 29:07When to use what?
- 33:49Introducing "Dataco"
- 37:33Create Tables in Hive
- 38:18Use Impala via Hue to Query
- 40:14Ingest Data Using Flume
- 41:17Create Hive Tables over Log Data
- 42:30802: Why is sales suddenly dropping?
- 44:28Flume and Morphline Pipeline
- 45:03Flume with Morphines Configured
- 46:14Takeaways
- 47:13Did you learn something?
- 49:06Dean Wampler
- 50:451 Map step + 1 Reduce step
- 52:49Inverted Index
- 56:59Problems
- 59:36Benefits
related talks
53mBig Data: from mammoth to elephant
47mSolving Real & Big (Data) Problems Using Hadoop
19mBig Data
45mSolving Real & Big (Data) Problems Using Hadoop
42mBuilding Robust Streaming Data Pipelines with Apache Spark
29mGrowing insights through hybrid data architectures
42mWeb Development To Big Data. A Journey
44mData Science at Scale with Apache Spark