Monitoring Serverless Workloads with OpenTelemetry and Prometheus
summary
Adds sidecars with lifecycle ordering to Cloud Run so collectors start before and stop after app containers, enabling reliable flushes; push-based OpenTelemetry flows without extra work, while Prometheus needs a scrape-on-shutdown final scrape and a modified Prometheus/OTel receiver that synthesizes start_time for cumulative metrics so single-scrape instances still report. Configs live in Secret Manager mounted as files and agents reload on change. Changes run in production on Cloud Run, reduce telemetry loss but increase CPU/memory overhead and require final scrapes to complete within 10 seconds.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:24Agenda and serverless basics
- 3:51When to use serverless with Cloud Run
- 5:18Serverless observability challenges
- 7:25Serverless telemetry problems
- 11:19Cloud Run observability changes
- 14:51Push-based metrics with OpenTelemetry
- 15:12Pull-based metrics with Prometheus
- 17:41Deploying collectors
- 19:27Future plans and contributions
- 20:07Current limitations
- 21:11Q&A
related talks
03mUsing OpenTelemetry and Prometheus to monitor serverless
05mOpenTelemetry in production for a year plus: the good the bad the ugly
30mOperating Prometheus in a Serverless World
34mFleeting Metrics: Monitoring Short-lived or Serverless Jobs with Prometheus
25mMonitoring in a Serverless World
06mLearnings from Running Prometheus on Google Cloud Scale
23mBeyond the Ephemeral: Mastering Serverless Metrics at Scale With Shopify
56mFull-stack observability and application monitoring with