YDB Topic Service: How We Improved Message Queue Performance
summary
YDB Topic Service is a message-queue component within the YDB platform that provides durability, scalability, ordering, and delivery guarantees. The talk explains performance improvements made to this Topic Service, describing the problems encountered with throughput and latency and the engineering approaches used to optimize the queue implementation.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:55What is YDB Topic
- 5:10Message queue architecture
- 7:17Kafka vs Pulsar
- 9:20Why left Kafka in 2017
- 13:00Kafka evolution and limits
- 18:42Why chose YDB
- 20:09Load testing method
- 23:00Pre-optimization results
- 25:09First optimizations and impact
- 27:41Profiling and bottlenecks
- 29:08Data encoding optimization
- 30:56Post-optimization results
- 33:28Future improvement plans
- 34:30New YDB Topic features
- 37:01Q&A
related talks
30mAn approach to unite tables and persistent queues in one system
44mYDB Java SDK: The Battle for Performance
51mComparing the Scalability of Kafka and YDB Topics
47mState machines for product data from YDB and queues
50mHow We Built a Distributed Queue Service at Yandex
38mTransactional Work with Topics
01h 00mParallel Asynchronous Replication b/t YDB Database Instances
51mYandex Database: Distributed Queries in the Clouds