Systems @Scale Summer Q&A Week 4
summary
Practical advice for debugging performance at scale: use BPF tracing on hosts to fill kernel metric gaps, export aggregated metrics to Atlas/Druid/Hive, and run targeted per-host tools when a bad instance is spotted; Netflix ships a TCP flow exporter and stores traces for analysis. GPU work uses Kineto and ASIC telemetry, saved in Chrome Trace format or a binary trace for multi‑view tools. Watch BPF overhead (don’t trace per‑packet on 1Mpps systems), prefer upstream BCC/libbpf tools, and use latency heat maps and PMCs to separate hardware, hypervisor (Nitro) and software causes.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:49Q&A format
- 3:18BPF for distributed traces
- 5:48Ad-hoc performance probing
- 8:37Workload movement tracking
- 10:02Resource limits in binaries
- 11:49Hardware for AI and video
- 13:34GPU training observability
- 15:59Cloud hosting performance
- 20:59Hardware vs. software debugging
- 23:54Resource sprawl management
- 27:45BPF for containers
- 30:12Software vs. hardware issues
- 32:23Tuning centralized systems
- 34:10BPF for distributed profiling
- 36:57Ensuring BPF scripts don’t cause performance issues
- 42:54BPF script maintenance
- 47:48AsicMon integration
- 48:53Open sourcing AsicMon
- 49:38BPF with monitoring
- 53:08AI performance debugging
- 54:11Sidecars for workloads
- 55:38AI processing trends
- 57:05Memory limits handling
- 58:52Closing remarks
related talks
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
26mProfiling Mastery: Decode Your Data and Elevate Observability
35mPerformance Insights into eBPF Step by Step
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
21mAuto-Instrumentation for GPU Performance using eBPF
44mNow You See Me Too: Visual Tooling for Advanced System Analysis
01h 01mThe Performance Investigator's Field Guide
23mNetwork Communication Debuggability and Observability at Scale