Observing Failure in Reactive Systems
summary
A logistics pipeline handling 100 million events/day uses Kafka topics, Cassandra persistence and a unique-identifier service; a compact Grafana dashboard surfaces message lag per partition and total, outstanding DB request queues, error-topic throughput, JVM GC frequency/duration and pod restarts. Observability catches a 14-hour stuck partition and repeated-file reingestion caused by middleware misconfiguration that flooded one topic, exposes Cassandra timeouts and circuit-breaker cascades, and drives adoption of correlation IDs and timeout tests so incidents become diagnosable and triageable.
AI-written summary · about
chapters (11)
- 0:00Observing failures in reactive systems
- 1:03Reactive system failures
- 3:29Logistics system architecture
- 5:17Key observability metrics
- 10:11Failure scenarios and fixes
- 15:47Timeouts in async systems
- 17:11File ingestion overload
- 21:23Poor observability impacts
- 23:52Observability benefits
- 25:16Lessons learned and future tips
- 26:22Q&A
related talks
33mObservability and Debuggability in Cassandra
33mObservability-First Kafka: Engineering Visibility at Scale
39mArchitectural design patterns applied to Distributed Systems
40mSurvivability of Observability: Taking your DIY system to success
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
01h 31mDeveloping Real-Time Data Pipelines with Apache Kafka
42mDistributed Monitoring: How to understand the Chaos
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale