DART: A Scalable and Adaptive Edge Stream Processing Engine
summary
DART implements a fully decentralized edge stream engine that uses a DHT-based overlay (Pastry) and runs on Apache Flume to place, chain and scale operators automatically; join messages route sources to sinks so operator chains form without a central master. Experiments show lower query latency and much faster deployment than Storm and EdgeWise when tens to hundreds of concurrent IoT queries run, and recovery uses ratio codes and leaf-set replicas to resume state quickly.
AI-written summary · about
chapters (10)
- 0:00Introduction to edge stream
- 1:37The DART engine design
- 2:40IoT use cases and challenges
- 3:44Scalability and architecture
- 6:03Handling edge dynamics
- 6:56Decentralized architecture
- 9:14Data flow construction
- 10:44Scaling and recovery
- 12:07Implementation and results
- 13:34Conclusion and future work
related talks
16mPreview: Cloud Computing
37mStorm, a Distributed Real-time Data Processing System
16mAStream: Ad-hoc Shared Stream Processing
38mSelf-managed and automatically reconfigurable stream processing
38mFaster and Furiouser: Flink Drift
41mLessons Learned Implementing a Resilient, Distributed, Real-Time Architecture
20mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer or Natural Outcome?
40mBrace yourself! Storm is coming!