A Kafka Client’s Request: There and Back Again
summary
Follows a Kafka client request from producer.send() through serialization, partitioning, batching and optional compression to the broker, where network threads hand it to IO threads that write to page cache (not immediate disk) and hold the request in purgatory until replicas ack — replication checks run about every 500 ms. Covers concrete defaults: linger.ms=0, batch.size≈16KB, buffer.memory≈32MB, max.request.size≈1MB, acks=all, network threads default 3 and IO threads default 8, and the metrics to watch (request-latency, localTimeMs, remoteTimeMs, totalTimeMs).
AI-written summary · about
chapters (19)
- 0:00Intro
- 2:05A Schema and a Topic
- 6:57Partitioning Configurations
- 12:20Batching Configurations
- 16:29Batching Metrics
- 21:00Request Configurations
- 24:04Request Metrics (General)
- 26:28Socket Receive Buffer
- 27:10Network threads
- 28:30Request Queue
- 29:54I/O Threads, aka Request Handler Threads
- 30:56Page Cache and To Disk
- 32:295. Page Cache/Disk Configuration and Monitoring
- 35:04Merder Purgatory (but actually)
- 36:50Response Queue
- 37:31Network thread hand-off
- 37:51Socket Send Buffer
- 38:24And back again to the Producer
- 39:55Start your own journey..
related talks
43mA Kafka Producer’s Request: Or, There and Back Again
05mIntroduction to Kafka Internals
53mDemystifying Kafka
48mKafka as a Message Queue: can you do it, and should you do it?
38mKafka Message Broker Under High Load
44mFive Years of the Good and the Not-So-Good with Kafka
46mKafka 4, Fantastic?
54mWhen Everything Went Kafka: Apache Kafka and the Consumer