Real Time Datamining and Aggregation at Scale
summary
Claims that Storm and Hadoop can cooperate to deliver sub-5 ms typical latencies and <5 s worst-case for a sample metrics app while keeping years of durable history. Demonstrates Mahout changes: a cleanup release plus new code for streaming k‑means that is single‑pass (≈20 μs/point at 10 dims → 1 minute per million), recursive centroid compression, and near‑linear parallel speedup to ~16 cores. Implements projection search and LSH XOR tricks to cut distance work; prototype hits within 2–4× of a bank customer’s target. Pig integration exists but is fragile.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:42Code Quality
- 3:05Fast Collections
- 4:08Math Library
- 4:42Future Releases
- 5:05KNearest Neighbor Modelling
- 9:07Hood Math Library
- 11:33Projection Search
- 13:54Kmeans Search
- 15:30Streaming Kmeans
- 18:56Parallel speedup
- 20:26The inner loop
- 22:33New stuff
- 22:54PigVector
- 24:26Tokenization
- 26:01Interactions
- 27:10Code
- 28:04Training Model
- 28:41Reservations
- 29:30Questions
- 30:22Pig
- 32:10Mahute
- 33:55Why not use Mahute
related talks
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
53mBig Data: from mammoth to elephant
12mthe Rapidly Changing Hadoop Ecosystem
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
17mA Real Time Analytics Framework
37mStorm, a Distributed Real-time Data Processing System
08mReal-Time Data Systems: Speed, Scale, and Resilience
42m"Distributed stream processing in Ruby"