The Long and Winding Road to Application Logging [A]
summary
Built a Cloud Foundry log ingestion pipeline using Elasticsearch/Logstash/Kibana (E/L/K) that first consumed user‑provided syslog endpoints, then switched to Loggregator Firehose for metadata and mass adoption, which reveals scale problems: field‑mapping errors in Elasticsearch, giant multi‑line stack traces treated as thousands of messages, and unpredictable message‑size spikes. The team adds rate limits, per‑app quotas and burst limits, provides structured JSON logging libraries (Java/Node) and finally implements a brokered forwarder called SLEEVE to enforce plans and retention; it works and will be open source.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 2:53Two-year journey overview
- 5:10Ingestion pipeline architecture
- 7:36Logger Gator subsystem
- 9:42User provided services start
- 13:39Switching to Logger Gator firehose
- 16:13Lessons from large-scale deployments
- 21:17Leaving syslog for service broker
- 23:24Pre-built dashboards demo
- 25:11Summary and takeaways
- 27:20Q&A session
related talks
29mCloud Foundry Logging & Metrics
46mBuilding a scalable logging platform
20mFast log management for your apps
48mLogging and ranting
31mLogging at Scale, Billion+ messages, 100K msg/sec
30mMonitoring Cloud Foundry: Learning about the Firehose
39mFast log management for your infrastructure
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution