Monitorama PDX 2024
| title | author | length |
|---|---|---|
| Logs Are Good, Actually | Alex Hidalgo | 27m |
Argues that plain logs remain essential: uses flight timestamps, local weather, conference GitHub history and text messages to reconstruct a real scheduling… | ||
| The Ticking Timebomb of Observability Expectations | David Caudill | 39m |
Argues that “monitor everything” leads to millions of low‑value signals, costly vendor bills and cognitive overload: expect 10k metrics per cluster, tracing… | ||
| The Hater's Guide To OpenTelemetry | Austin Parker | 38m |
OpenTelemetry frames itself as a standards‑style schema for metrics, traces, logs and profiles, promising vendor‑agnostic telemetry and OTLP traces from Docker… | ||
| Things I wish I knew before we decided to migrate our metrics infrastructure.. | Suman Karumuri | 28m |
Migration moves seven things at once: instrumentation from statsD to OpenTelemetry, collection (OTel collector sidecars with a patched statsd receiver), a… | ||
| The Observability Data Lake, 1 year on | David Gildeh | 31m |
Netflix runs an Observability Data Lake in production ingesting about 160 billion traces (≈1 trillion spans) and three petabytes of logs per day, storing data… | ||
| The subtle art of misleading with Statistics | Dave McAllister | 30m |
Shows concrete tricks and real numbers: switching between arithmetic vs harmonic mean turns 1.56 hours into 0.53 hours; two samples with SD 3.3 ms and 44.8 ms… | ||
| Is Your Kernel Being Honest? Understanding & measuring low level bottlenecks | Ferris Ellis | 32m |
A short demo program allocates 1 GB but pmap shows 1 GB virtual maps to 4 KB RSS, and touching that memory triggers ~250k page faults so that populating the… | ||
| The complexity of success and failure: the story of the Gimli Glider | Thai Wood | 30m |
A detailed retelling of the Gimli Glider shows how a unit conversion in kg vs volume combined with a taped-out FQ channel and misleading gauges leads to taking… | ||
| Pugs, Poe's and pipelines; An engineering perspective on big-data streams... | David Josephsen | 33m |
Argues that early, local reduction of telemetry cuts latency and cost, introducing Point of Enrichment (PoE) and Point of Aggregation (Pug) as primitives. Uses… | ||
| How we tricked engineers into utilizing distributed tracing | Noa Levi | 29m |
Strava migrates from OpenTracing to OpenTelemetry, asking each team to bump a shared library, deploy, and test about 100 microservices themselves; that… | ||
| Experiments in Backing Prometheus with Clickhouse | Colin Douch | 28m |
Builds a Remote Write/Read bridge that inserts Prometheus samples into ClickHouse and replays a week of real queries against 24 hours (~850 GB) of production… | ||
| From Alerts to Insights: Performing Trace-Based Causation at Scale | Logan Rosen | 31m |
Engineers build a streaming pipeline that reconstructs traces in memory from span messages, Go and Kafka power the consumer, and per-trace depth‑first searches… | ||
| From Polling to Streaming: Network Monitoring with Real-Time Telemetry | Tushar Gupta | 30m |
Explains why SNMP polling fails at scale and shows a working replacement: push-based streaming telemetry using gnmi over gRPC with OpenConfig models to… | ||
| Serverless Observability: a case study of SLOs | Virginia Diana Todea | 31m |
Elastic migrates observability to Kubernetes by building SLOs on transforms that summarize data and scale across clusters; transforms roll up to one document… | ||
| Distributed Tracing - All The Warning Signs Were Out There! | Vijay Samuel | 33m |
eBay runs tracing at ~400 billion spans/day (about 5M spans/sec), 3,000 CPUs and ~200 TB for 14‑day retention, using OpenTelemetry and ClickHouse sharded into… | ||
| Welcome to the OTEL California | Matt Macdonald-Wallace | 26m |
30-minute, hands-on account of using OpenTelemetry with Python and Django showing what actually works and what doesn’t: install the SDKs and exporters, run a… | ||
| Tracing Service Dependencies at Salesforce | Sudeep Kumar | 29m |
Salesforce processes ~300 million spans per minute and ~100 million unique traces per minute through a Flink-based streaming pipeline that builds in-memory… | ||
| Distributed Context Propagation: How you can use it to Improve Observability.. | J. Kalyana Sundaram | 31m |
Standardizes a single HTTP header for propagating arbitrary key/value request context so services can do traffic prioritization, targeted chaos or fault… | ||
| Use counters to count things | Fred Moyer | 25m |
Shows a 32‑bit counter overflow hitting exactly 2,147,483,647 rollover and fixed by switching to string formatting, demonstrates a costly log search over… | ||
| The shoemaker’s children have no shoes - why SRE teams must help themselves | Pete Fritchman | 28m |
Shows how an SRE team fixes a neglected internal CI environment called testbox by adding metrics and traces (a simple testbox tracing via log_event approach)… | ||
| Incident Management: Lessons from Emergency Services | Julia Thoreson | 35m |
Compares incident workflows between a platform serving reporters and a missing-hiker search, showing concrete practices: decouple alerting and notification so… | ||
| No observability without theory | Dan Slimmon | 25m |
Claim: more telemetry alone doesn’t buy observability; interpreting outputs requires a maintained theory of system behavior. Practical moves: design dashboards… | ||
| Disintegrated telemetry: The pains of monitoring asynchronous workflows | Johannes Tax | 32m |
Asynchronous, temporally decoupled workflows create gaps that tear traces into chunks, so simple root-span tracing breaks at scale. The practical solution uses… | ||
| The Hater's Guide to Dealing with Generative AI | Phillip Carter | 29m |
Claim: production generative systems succeed or fail on observability not model magic. Built a natural-language→Honeycomb query feature using gpt-3.5 in… | ||
| nothing matches | ||