Understanding Performance w/DTrace (While The Customer Yells)...
summary
DTrace provides dynamic, production-safe instrumentation (about 400,000 probes on a Mac) and lets operators iterate with aggregations and quantize primitives to ask precise questions. A customer NFS workload shows a tri-modal latency up to 2s for individual writes, while back-end disks look fine. Tracing exposes a hot lock causing 24% CPU and threads descheduled for seconds, plus a brittle write-throttle; the team rewrote the write-throttle, tuned queue depth to share pain with storage, and collapses the bad distribution into a single, acceptable mode.
AI-written summary · about
chapters (11)
- 0:00Start
- 19:12DTrace Summary
- 22:52Customer System
- 33:27sync write time contribution
- 34:45Basic Performance Goals
- 36:51How ZFS Processes Data
- 40:32Increasing Queue Depth
- 43:13Profiling
- 44:07Lockstat
- 45:53Happy Ending
- 47:04Lessons Learned
related talks
11mFinding out what's really going on, with DTrace!
37mDiving into the Details with DTrace
35mTracing is More Than Traces: The Insights in Trace Aggregates
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
01h 01mThe Performance Investigator's Field Guide
05mPerformance analysis using Systrace
30mDTrace at 21: Reflections on Fully-grown Software
37mPerformance Diagnostics with Open Distro for Elasticsearch