Deploying Kafka on DC/OS
summary
Practical checklist for running Kafka on DC/OS: deploy three brokers with one broker per host using DC/OS placement constraints and pinned volumes, give brokers dedicated disks (prefer HDDs for throughput, SSDs only selectively), allocate modest JVM heap (1–2 GB) but large page cache (dozens of GB), run Zookeeper as a separate quorum (don’t share with Mesos masters), perform rolling restarts with gaps, treat Kafka Streams apps as stateful and allow time to rebalance. Confluent supplies a DC/OS package (Confluent Platform + Kafka CLI docker images) that implements these patterns.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:18About Confluent
- 2:11Confluent Platform and Apache Kafka
- 3:19What is Apache Kafka?
- 5:31What's in a Kafka Cluster?
- 8:14Why Kafka on DC/OS?
- 9:43Kafka Cluster Services
- 10:32Kafka Services are Stateful
- 11:27How do you manage all these?
- 12:11Where to place these services?
- 14:05DC/OS benefits for Kafka
- 17:04Service config sample
- 17:55Broker config sample
- 18:26Kafka Storage Volumes
- 22:07What about Zookeeper?
- 24:55Kafka Broker Restarts
- 25:43Kafka Streams with fault-tolerance
- 28:38Failures
- 32:02Other useful CLI tools
related talks
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
38mRe-architecting Apache Kafka for the Cloud
16mIntroducing Apache Kafka
53mDemystifying Kafka
05mIntroduction to Kafka Internals
33mManaging a Kafka cluster in KRaft mode with Strimzi
30mWhy Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design