Apache Flink meets Apache Mesos and DC/OS
summary
Apache Flink runs as a Mesos framework using a Fenzo scheduler for offer-to-task matching, a Flink Mesos dispatcher for high availability, and per-job JobManagers to enable dynamic, resource-elastic clusters; integration is packaged for DC/OS and deployable with DC/OS package install flink. A Sept 2016 survey shows ~30% of users already ran Flink on Mesos; Mesos claims 10,000+ node scale. Demo deploys Kafka and Flink on a 4-node cluster (16 cores, 55 GB RAM), processes transactions with a 10-minute window sliding every 2 minutes and flags totals over €10,000 — the demo successfully detects suspicious transactions.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:06Data center evolution
- 3:39Event processing stack
- 5:05Resource utilization problem
- 6:13Apache Mesos benefits
- 8:26Flink Mesos integration
- 10:57Integration components
- 13:12Component interaction
- 14:58Fenzo scheduler
- 16:24Dynamic scaling plans
- 18:57Flink Mesos architecture
- 20:24DC/OS overview
- 22:12Demo setup
- 28:16Deploying services
- 30:08Submitting Flink job
- 32:24Demo results
- 32:45Conclusion summary
- 33:32Q&A session
related talks
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
42mApache Flink deep-dive #bbuzz
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
20mAutoscaling Apache Flink Applications
59mBuilding Distributed Systems with Apache Mesos
32mFlink architecture and automation in Intesa Sanpaolo
36mApache Flink in the Cloud-native Era
30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?