Performance Analysis of Cloud Applications
summary
Gmail production traces show requests involve tens to thousands of RPCs across a billion accounts and load and request mix change continuously. Two practical techniques collect actionable data: vertical context injection embeds high‑level labels into kernel traces (ftrace + system‑call tagging) so kernel events map to specific operations and Dapper RPC spans, and coordinated bursty tracing uses a bitmask on millisecond clocks to sample bursts across processes. At tens of millions of users per partition this detects ~1% regressions, kernel buffers hold ~10s of trace and overhead stays under 3%, and an injection trace once pinpoints a thundering herd of tiny attribute reads causing CPU bursts and throttling.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:20Changing Cloud Applications
- 0:59Representative Load
- 1:09Queries Per Second
- 2:32Average Response Size
- 3:17Why is this happening
- 4:16CPU Usage Graph
- 5:42User Visible Requests
- 6:37Non User Visible Requests
- 8:41Scale Latency
- 10:26Operational Context
- 13:25Vertical Context Injection
- 16:58Temporal Context
- 17:28Bursty Tracing
- 19:42Summary
- 20:47QA
related talks
01h 00mSystems @Scale Summer Q&A Week 4
11mFinding out what's really going on, with DTrace!
54mUnderstanding Performance w/DTrace (While The Customer Yells)...
25mPerfect Match: Correlating Continuous Profiling with Distributed Tracing for Stronger Observability
31mParting the Clouds: Tracing Cloud Foundry for Improved Performance...
52mTracking Down What Happened: A Comparison of Tracing Solutions
17mEnd-2-End Resource Accounting Leveraging Distributed Tracing
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...