Systems @Scale Summer 2021
2021-06-09..2021-06-30
-- online --
[Systems @Scale Spring 2021, Systems @Scale Summer 2021, Systems @Scale Fall 2021, Systems @Scale Winter 2021, Systems @Scale Summer 2022, Systems @Scale Winter 2022, Systems @Scale Summer 2023, Systems @Scale 2024]
who it's for
site reliability engineers, platform engineers, cloud infrastructure engineers, kubernetes operators, backend engineers, storage engineers, data engineers, observability engineers
topics
load shedding and retry control; circuit breaker SDKs; adaptive concurrency limits; hedge requests and pick-two load balancing; large-scale traffic load testing; autoscaling and backpressure; shuffle sharding for isolation; managing millions of Kubernetes clusters; dynamic leasing of spare capacity (Optimus); co-location of mixed workloads; datacenter-spanning filesystem consolidation (Tectonic); key-value engine design (ArkDB)
| title | author | length |
|---|---|---|
| Avoiding Overload in Distributed Systems | David Yanacek | 25m |
Shows concrete ways to keep goodput flat under overload: load shedding using CPU/latency/connection heuristics and an SDK circuit breaker to stop harmful… | ||
| Better Resilience Through Self Tuning | William Thurston | 20m |
Netflix-style control plane uses adaptive concurrency limits, hedge requests, and choice-of-two load balancing to keep services responsive. The system handles… | ||
| Live Traffic Load-Testing-Measuring and Validating Capacity at Facebook | Kiryong Ha, Lin Xao | 21m |
Facebook runs 80,000 tests per day across ~250 services and consumes one-third of capacity to measure maximum servable load; service tests ramp traffic until… | ||
| Defining Service Level Objectives for Exposure Notifications Servers | Marilia Melo | 17m |
Defines an availability SLO initially set to 99.5% SLO using the Google Cloud Load Balancer https request count metric, then narrows scope to the user-facing… | ||
| Systems Scale Summer Q&A Week 1 | Michael Chang | 35m |
Q&A summarizes concrete practices: Netflix open‑sources Netflix concurrency limiters and uses a runtime monitor that auto-tunes per‑host limits; load balancing… | ||
| Managing a Million Kubernetes Clusters | Qi Ke | 26m |
Azure Kubernetes Service runs and troubleshoots a million clusters by adding telemetry, automated detection and remediation, and tighter guardrails rather than… | ||
| Morcor - Co-Location of Mixed Workloads at Uber | Amite Bose | 22m |
Uber co-locates stateless microservices and preemptible batch jobs by dynamic partitioning and host overcommitment, running containerized YARN NodeManagers on… | ||
| Dynamic Leasing of Spare Capacity to Improve Fleet Utilization with Optimus | Ariel Rao | 26m |
Optimus is a production platform that leases idle servers to other teams by combining a resource broker, reservation database and allocator with a supply… | ||
| Systems @Scale Summer Q&A Week 2 | Yun Jin | 43m |
Kubernetes is evaluated alongside Mesos and Yarn with concrete tradeoffs: GKE tops at ~15,000 nodes but operators prefer many 1,000 node clusters for fault… | ||
| What is Behind Alibaba's Double 11 Shopping Festival | Yuan Mei | 23m |
Flink (Blink improvements) powers Tmall’s Double 11 dashboards, processing peaks of about 4 billion records/sec and ~7 TB/s with end-to-end latency under one… | ||
| Consolidating Storage Backends at Scale with Tectonic Filesystem | Satadru Pan | 19m |
Consolidates Haystack, f4 and HDFS into a single datacenter-spanning filesystem called Tectonic that stores multiple exabytes per cluster and runs as a… | ||
| ARKDB: The Key-Value Engine for Alibaba Cloud Storage Services | Nathan Pang | 24m |
ArkDB is a key-value engine built for Alibaba Cloud on top of Pangu that combines LSM ideas with Bw-tree–style page mapping to reduce compaction impact and… | ||
| Systems @Scale Summer Q&A Week 3 | Hong Tang | 35m |
Two storage stacks are compared by implementation and tradeoffs not marketing: one (RTD/arcdb) is rebuilt for NVMe and fast datacenter networks, uses a… | ||
| Performance Wins with BPF: Getting Started | Brenden Gregg | 21m |
Installs the BCC and bpftrace toolsets, then runs lightweight probes like execsnoop, opensnoop, tcplife and biosnoop for fast wins. Example: iostat shows… | ||
| Platform Agnositc Observaility System for AI Accelerators | Brian Coutinho | 18m |
Facebook reports running about 150 trillion AI predictions per day and processing 250 million video uploads daily. It builds an observability stack: asicmon… | ||
| Resource Guarantees in a Multi-Tenant Infrastructure | Pavel Paulau | 18m |
Facebook builds a pipeline that measures per-process CPU and memory across the fleet, detects regressions as small as ~15 MB memory or 0.03% host CPU, and… | ||
| Systems @Scale Summer Q&A Week 4 | Yun Jin | 01h 00m |
Practical advice for debugging performance at scale: use BPF tracing on hosts to fill kernel metric gaps, export aggregated metrics to Atlas/Druid/Hive, and… | ||
| nothing matches | ||