A Crash Course in Error Handling for Streaming Data Pipeline
summary
Covers concrete patterns for Kafka-based pipelines: use retry with exponential backoff for transient network or broker failures and set retries>0 and retry.backoff.ms in clients; detect memory leaks with Kubernetes liveness/readiness and restart on limits; route malformed records (Avro vs JSON deserialization failures) to a dead-letter queue by configuring Kafka Connect or a custom Kafka Streams exception handler that produces failed records and metrics; monitor consumer lag and DLQ producer rate via JMX and alert, and assign clear ownership for manual fixes.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 1:02What is a streaming pipeline
- 4:13Challenges in error handling
- 5:41Types of errors in pipelines
- 8:55Handling transient errors
- 11:28Handling non-transient errors
- 14:41Dead letter queues for errors
- 22:30Monitoring pipeline health
- 26:00Detecting back pressure
- 30:12Key takeaways and summary
related talks
46mMake Your Kafka Applications More Robust
41mPatterns and anti-patterns for production ready Kafka Streams apps
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
27mNavigating Streaming Infrastructure
50mStream processing with Kafka in Big Data environments
38mInteractive Session: Optimising data streaming pipelines on Flink and Kafka
01mScale By the Bay 2021 intro video by Opher Dubrovsky
50mApplication development and data in the emerging world ...