Demystifying Big Data
summary
Defines big data as a scalable pipeline of generation, transformation, storage, processing and analysis and insists the real test is business impact not technology. Describes running 2.2+ trillion events a year for 1+ billion users with ~44 billion programmatic ad requests and ~86 million Hadoop jobs, choosing open source stack and deploying mini Hadoop clusters across geographies to avoid central bottlenecks. Shares a 78% queue reduction example and stresses build-for-impact over buzz.
AI-written summary · about
chapters (9)
- 0:00What is big data
- 3:55Data replaces oil
- 6:44Sensors and IoT revolution
- 9:16Google Maps and real-time data
- 11:01ATM withdrawal optimization
- 12:05Myspace and Friendster failures
- 12:47Big data infrastructure choices
- 16:59Open source contributions
- 17:42Global data processing challenges
related talks
57mBig Data: Threat or Opportunity?
46mA Brief History of Big Data
45mSolving Real & Big (Data) Problems Using Hadoop
47mSolving Real & Big (Data) Problems Using Hadoop
56mResistance is Futile: What You Need to Know about Big Data
08mBig Data and the Fact-Based Enterprise
29mBig Data Analysis
36mSmall intro to Big Data