Resource Guarantees in a Multi-Tenant Infrastructure
summary
Facebook builds a pipeline that measures per-process CPU and memory across the fleet, detects regressions as small as ~15 MB memory or 0.03% host CPU, and rolls changes through canary experiments (≈80 test vs 80 control hosts) before rollout. Resource accounting and isolation use systemd + cgroups and eBPF for profiling, plus jemalloc-based memory metrics and a container pressure tool to produce working‑set profiles; the system is deployed and reduces contention while freeing capacity.
AI-written summary · about
chapters (9)
- 0:00Resource guarantees intro
- 2:51Resource contention origins
- 5:40Data source for accounting
- 7:27Resource monitoring pipelines
- 9:31Regression prevention with canaries
- 11:34Accounting with cgroup hierarchies
- 13:24Memory accounting with Senpai
- 15:08Recovery from unsafe interactions
- 16:15Future efficiency vision
related talks
34mThe Missing Metrics: Measuring Memory Interference in Cloud Native Systems
35mResource Management Ecosystem and Cloud Foundry - A Review of the State of the Art
17mEnd-2-End Resource Accounting Leveraging Distributed Tracing
33mCloud Foundry Runtime PMC Overview - Dieu Cao, Pivotal
34mEfficient Resource Management
27mWhen Linux Memory Accounting Goes Wrong
10mCost Optimization in Cloud Foundry Through Economies of Scale
46mLessons learned from monitoring a large production network