End-to-End Encryption for Streaming Data Pipelines
summary
Kryptonite for Kafka provides client-side field-level cryptography for Apache Kafka, implemented as a custom SMT for Kafka Connect that encrypts selected JSON fields (arrays, nested objects supported) at source and a set of Flink SQL UDFs to decrypt/re-encrypt during stream processing. Demo pipeline uses MongoDB via Debezium, the SMT encrypts SSN/addresses, keys live in container files for the demo with Azure Key Vault support planned, Flink job computes BMI and writes encrypted full_name back to Kafka, and sink to MinIO (S3) successfully outputs decrypted records. The Flink UDFs are experimental; repo is open source.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 3:30Limitations of standard Kafka security
- 5:37Introducing Kryptonite for Kafka
- 8:28Demo: MongoDB to Kafka with encryption
- 12:21Demo: Stream processing with Flink
- 17:21Demo: S3 sink with decryption
- 19:08Project overview and key management
- 21:35Q&A: Schema and multi-producer scenarios
related talks
51mClient-Side Field-Level Encryption for Apache Kafka Connect
45mData Privacy for SaaS Kafka or Pulsar Offerings
40mKeeping data private in real-time pipelines
45mDeep Dive into Data Streaming Security
30mThe Secure Metamorphosis: Streaming Logs with Kafka and TLS
30mFlink Security Enhancements
07mDescribing Kafka security in AsyncAPI
27mEncrypting 6 Petabytes of Customer Data with AWS KMS