Multi-tenancy in Apache Kafka, Navigating a Major Topic
summary
Apache Kafka multi-tenancy at large scale: a protocol proxy gateway rewrites metadata and broker responses to present per-tenant virtual clusters, intercepts produce/consume requests to enforce policies and transformations, and offloads large files to S3 with caching to reduce network and partition costs. An open SDK of interceptors implements rules such as ACLs, retention and partition limits; a demo shows isolated topics and live policy rejections in a real deployment.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:06What is multi-tenancy?
- 9:16Tenant definition in Kafka
- 12:22Multi-tenancy challenges
- 18:00Possible solutions and limits
- 21:15Kafka proxy usage
- 27:33Message interceptors and assertions
- 31:38Security and best practices
- 33:02Handling large files
- 34:51Proxy cost benefits
- 36:15Partition reduction and optimization
- 37:21Proxy global advantages
- 40:07Data majority in Kafka
- 41:52Conductor Gateway open source
- 43:38Q&A
related talks
42mPartial Multi-Tenancy on Kafka Using Strimzi at LittleHorse
38mRe-architecting Apache Kafka for the Cloud
41mApache Kafka simply explained
47mPainless Multi-Tenant Kafka on Kubernetes with Istio at ASML- 47mPainless Multi-Tenant Kafka on Kubernetes with Istio at
50mIntroduction to Apache Kafka
50mMultitenancy from scratch
53mDemystifying Kafka