Message processing failed! But what's the root cause?
summary
Shows how to find root causes in message-driven systems by combining modeling, integration tests, and instrumentation with OpenTelemetry; uses the AAA test pattern but asserts outgoing messages and exact message properties. Demonstrates .NET Activity-based spans, attaching tags/events, and exporting traces to tools like Datadog or Jaeger. Explains cross-service propagation via the traceparent header (W3C tracecontext), connecting trace IDs to logs, and searching failed traces to locate a mis-mapped order ID that caused product not found errors.
AI-written summary · about
chapters (18)
- 0:00Introduction and setup
- 1:05Distributed system evolution
- 3:54Message-driven system interactions
- 5:20Debugging message failures
- 11:06Testing in distributed systems
- 13:17Production failures and root causes
- 15:23Distributed tracing necessity
- 17:12OpenTelemetry and standards
- 19:41Observability explained
- 20:02Traces and spans
- 23:14Trace propagation
- 24:38Getting started with OpenTelemetry
- 26:44Key tracing questions
- 28:49Tracing in C# code
- 31:01Traces vs logs
- 32:28Benefits beyond failures
- 34:37Recap and call to action
- 35:41Q&A and resources
related talks
58mMessage processing failed! But what's the root cause?
01h 13mMessage Processing Failed! But What’s the Root Cause?
55mMessage processing failed! But what's the root cause?
32mDisintegrated telemetry: The pains of monitoring asynchronous workflows
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
41mDistributed Tracing in .NET
47mI'm Tracing You!
13mSparking Joy for Engineers with Observability