Comparing Streaming Data Options For Speed, Simplicity & Safety

on YouTube ↗24m

" summary

Benchmarks use the OpenMessaging Benchmark on AWS i3en instances and compare Kafka and Redpanda at 50MB/s, 500MB/s and 1GB/s with TLS/SASL on. Redpanda (C++ per-core, no JVM) hits p99.99 ≈ 17ms at 500MB/s on 3 nodes; Kafka cannot complete on 3 nodes, requires 4–9 nodes for 113–389ms tail latency and shows >6s tails at 1GB/s on 6 nodes. Redpanda uses tiered S3 storage and Raft (no ZooKeeper), cutting production node count (18→3) and claiming ~$187k vs $1.2M yearly TCO (≈7× savings).

AI-written summary · about

" chapters (10)

1. 0:00Introduction to streaming options
2. 0:24Four eras of messaging history
3. 3:29Kafka's dominance and ecosystem
4. 5:35Introducing Redpanda
5. 8:28Benchmarking with Open Messaging
6. 12:03Latency comparison results
7. 14:13Cluster sizing and retention
8. 17:24Cost comparison breakdown
9. 20:14Redpanda deployment options
10. 23:22Q&A and resources

" related talks

• 01h 28mHigh Performance Stream ProcessingStephane Maldini, David Turanski, Glenn Renfo · SpringOne2GX 2015
• 38mData Pipeline with Apache KafkaMoritz Siuts, Robert von Massow · code.talks 2015
• 30mFive Challenges in Real-time Stream Processing and Five SolutionsFawaz Ghali · SREday 2023 London
• 39mEvaluating Streaming Framework Performance for a Large-Scale...Ron Crocker · Flink Forward Berlin 2016
• 44mData Streaming: Which horse to bet on?Kamil Owczarek · GeeCON 2019
• 32mShould you read Kafka as a stream or in batch? Should you even care?Opher Dubrovsky, Ido Nadler · Berlin Buzzwords 2021
• 50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by DesignRanganathan "Ranga" Balashanmugam · Great Indian Developer Summit 2015
• 20mAnalyzing streaming data with Apache DruidDarin Briskman · AWS re:Invent 2023