Configuring Kafka Connector to Process Typed Data and Handle Errors

on YouTube ↗14m

" summary

Shows how to configure Kafka Connector 5.15 to process typed data using JSON Schema, Protobuf and JSON converters (JSON converter embeds the schema) and to use new payload.mode compact/extended (compact is simpler but breaks on property type changes and lacks temporal and Protobuf support). Error handling uses Kafka Connect properties like errors.tolerance=none vs all, errors.deadletterqueue.topic.name and headers.enable; demo builds a sink connector, sends a bad message to the DLQ, shows connector fails with none and stays running with all.

AI-written summary · about

" chapters (11)

1. 0:00Introduction to Kafka Connector
2. 0:31Agenda and Type Data Basics
3. 1:59Schema Converters and PayAsYouGo Mode
4. 3:03Compact vs Extended Modes
5. 3:45Error Handling in Kafka Connector
6. 5:39Demo Setup and Environment
7. 7:31Creating the Connector
8. 9:01Successful Data Processing
9. 10:08Handling Failed Messages
10. 11:38Error Tolerance Configuration
11. 13:09Conclusion and Q&A

" related talks

• 05mPREVIEW: Kafka Connect Best PracticesRandall Hauch · Kafka Summit San Francisco 2017
• 42mFrom zero to hero with Apache Kafka's connect APIRobin Moffatt · Berlin Buzzwords 2019
• 38mA Crash Course in Error Handling for Streaming Data PipelineStefan Sprenger · Berlin Buzzwords 2023
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 46mMake Your Kafka Applications More RobustStephane Derosiaux · Devoxx France 2024
• 38mStreaming with StructureKatherine Stanley · GOTOpia February 2021
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 45mFrom Zero to Hero with Kafka ConnectRobin Moffatt · Voxxed Days Microservices 2019