Data Pipeline with Apache Kafka
summary
Running Apache Kafka on AWS: measured a six-broker C4xlarge-on-EBS cluster handling about 2,600 messages/sec per broker (~15,600 total) using topics with replication factor 2, 14-day retention and 20 partitions. Found gzip gives ~50% size reduction (snappy ~10%), Zookeeper experiences high IOPS from frequent consumer-offset commits, and slow EBS or network saturation causes replication lag (ISR lag ~4,000 msgs) leading producers to block/back off and risk dropped messages; a Raspberry Pi demo showed live leader re-elections and observable data loss when brokers are removed.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:39Who we are
- 1:05About emetria
- 6:13Kafka Producer
- 7:11What Kafka does with the data?
- 8:11API High Level Consumer
- 10:01emetriq Architecture
- 14:53TTL does not apply during Broker Recovery
- 16:03Scaling/Performance
- 18:03Slow EBS
- 18:49Producer Queues
- 20:15Our Kafka Cluster
- 20:49Fractals via Kafka
- 32:11What else you want to know
related talks
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
38mKafka Message Broker Under High Load
24mThe Cost of Apache Kafka
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
32mScaling your Kafka streaming pipeline....
01h 00mPutting Apache Kafka to Use
33mScaling your Kafka pipeline can be a pain - but it doesn’t have to be
43mBuilding Scalable and Extendable Data Pipeline