Addressing Delivery Reliability in Loggregator
summary
Loggregator replaces UDP and websockets with gRPC inbound to Metron and mutual TLS, adds a diode ring buffer between components, moves discovery off Etcd to Bosch DNS, and splits syslog drains into a scalable adapter/reserve-proxy flow. Monitoring uses blackbox tests every 15 minutes: 2 logs/s, ~1,000 logs/s, and a stress flood around 2–4 million logs/sec with a 70/20/10 weighting; results show dramatic reliability gains after the gRPC upgrade (CF release 253) but CF logs recent latency and the CLI 15s timeout remain unresolved.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:51Today's Agenda
- 1:58There are 2 hard problems in Loggregator.
- 4:26Loggregator User Experiences
- 6:25Loggregator system.
- 10:37Doppler - HA Routing Mechanisms
- 14:36Introducing Scalable Syslog
- 17:28Service Level Indicators & Objectives
- 20:26Why whitebox monitoring is so hard.
- 21:33Loggregator Black Box Testing
- 23:37Blackbox Message Reliability
- 26:32Blackbox - Logs Recent Latency
- 27:46Release Science
- 29:37Loggregator Tactical Roadmap
related talks
32mTuning and Managing Loggregator, the Cloud Foundry Logging Subsystem
31mDefining Service Level Objectives for Loggregator
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
29mCloud Foundry Logging & Metrics
30mMonitoring Cloud Foundry: Learning about the Firehose
22mEverything You Need to Know About Syslog Drains - Hands on Edition
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec