Building a Big Data Collection and Processing System
summary
A system for collecting, storing and processing massive live video feeds (petabytes/day) is presented, using Apache Mesos as the common runtime to run both Apache Hadoop and the video-processing compute cluster. The design uses containers to automate deployment and management of Hadoop, Mesos, and the various processing components for a municipal-scale surveillance use case.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:00Project context
- 8:00Collection architecture
- 14:53Processing challenges
- 20:40Performance optimization
- 25:38Pipeline demo
- 30:24Future work
- 34:24Q&A
related talks
36mHow to Build a Solution Processing Terabytes of Data (...)?
53mBig Data: from mammoth to elephant
57mBig Data: Threat or Opportunity?
49mBuilding a 100‑Petabyte Analytics Warehouse
44mProcess and analyze millions of events per second with open source tools
39mReal-Time Event Processing and Handling Stateful Applications on Apache Mesos
33mDesigning Architectures: Big Data + Real Time
19mDemystifying Big Data