Real-/Neartime analysis with Hadoop & VoltDB
summary
Combines Hadoop MapReduce for batch with VoltDB in‑memory for streaming to handle many small events and deliver near‑real‑time analytics. Pipeline moves raw logs out of DFS for batch jobs and routes event streams into VoltDB, avoiding heavy HDFS writes; claimed end‑to‑end latencies around 50 seconds and ability to absorb bursts (examples: ~1,000 events in five minutes, scaled tests up to ~500,000 events). Built a horizontal, fault‑tolerant ingest and query layer that reports success in prototype deployments.
AI-written summary · about
chapters (6)
- 0:00Introduction and company background
- 3:50Real-time analysis requirements
- 7:15Initial challenges with Hadoop
- 10:10Switching to VoltDB for scalability
- 13:49Architecture improvements and benefits
- 16:29Conclusion and future outlook
related talks
32mBuilding Real-time Systems with Open Source Technologies
42mScaling ACID SQL
29mEnhancing Vector Databases with Real-time Stream Processing
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
14mMIT Impact
49mData Platform Architecture, Evolution, and Philosophy
48mHyper Speed! When Big Data Blooms
45mSolving Real & Big (Data) Problems Using Hadoop