Fallout: Dead Letter Management At Scale
summary
A production AWS SQS-based DLQ manager monitors 50+ queues handling about 20 million messages/month, automatically retries DLQ items on schedules (hourly or 15‑minute) and tracks DQ count in a message attribute so retries and age are visible, supports per-message or bulk purge and batch move (up to 10 messages per call) to save time and cost, restores W3C trace context and links trace IDs to DataDog logs for root‑cause inspection, and adjusts visibility/consumption behavior to inspect without consuming.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:10Dead letter queues matter
- 6:01Why messages fail
- 10:57DLQ support today
- 12:19Essential requirements for managing dead letters
- 16:53Self-healing DLQ system
- 21:04Live demo
- 30:20Implementation details
- 37:25Idempotency and ordering
- 43:01Message handling tips
- 45:26Tracing and compatibility
- 48:37Summary
related talks
46mReal-world Examples of Message Queue Usage: Insights from RabbitMQ
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
01h 03mChoosing the Right Messaging Solution for Your Architecture
42mIt's like 10,000 streams when what you need is a queue
40mApache RocketMQ - Trillion Messaging in Practice
53mMaking tomorrow's code look like today's
50mMiddleware Showdown: Exploring Diverse Messaging Solutions
30mThe last in the line: a talk about DevOps and Disasters