Concord: Simple & Flexible Stream Processing on Apache Mesos
summary
Concord is a Mesos-integrated stream processor with a core in C++, operator isolation via Mesos containers, per-operator RocksDB local state, and three routing modes (shuffle, round-robin, group-by with consistent hashing). It samples traces at about 1/1000 messages and exports to Zipkin, runs multi-language operators (Python/Ruby/Go/Java/Scala/C++), and claims consistent performance across seven job types with p90/p99 ≈ 10ms; current delivery is best-effort processing while Kafka-backed exactly-once is planned.
AI-written summary · about
chapters (8)
- 0:00Introduction to stream processing
- 3:24Challenges with Apache Storm
- 7:43Concorde architecture overview
- 12:56Performance benchmarks vs Storm and Spark
- 15:47Message processing guarantees
- 17:14Future development plans
- 19:19Use cases and recommendations
- 20:25Q&A session
related talks
42mStream Processing with Apache Flink
41mWhat If We've Been Scaling Stream Processing Wrong All Along
42mMesos: The Operating System for your Cluster
42mPrinciples and Patterns for Streaming Data Analysis
39mHow we learned to love the Data Center Operating System
15mMonoceros: Faster and Predictable Services through In-pod...
59mBuilding Distributed Systems with Apache Mesos
34mStream Processing at the Edge with Apache Kafka