Real-Time Adaptive Controls for Resilient Distributed Systems
summary
CrowdStrike adapts hundreds of tunables in real time for Falcon by sampling errors and latencies from Kafka/Cassandra paths and inferring limits without central coordination. Implementations include a percentage-threshold circuit breaker, EWMA backoff driven by an error reactor with weighted failure modes, and adaptive concurrency via AIMD (with planned TCP Vegas delay‑based control). Deployed on services handling 50M concurrent agents and ~2 trillion events/day (one writer at ~85M req/s), yielding ~40–50% compute reduction and stable production rollout.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:01Team and responsibilities
- 2:44Need for adaptive controls
- 4:10Reference architecture
- 5:11Adaptive circuit breakers
- 8:33Adaptive backoff strategies
- 12:14Dynamic concurrency control
- 18:47Adaptive rate limiting
- 23:21Adaptive autoscaling
- 29:45Future roadmap and research
- 34:35Dynamic cache management
- 36:12Conclusion and Q&A
related talks
18mReal-time adaptive controls for resilient container workloads
20mBetter Resilience Through Self Tuning
37mRethinking How We Test Our Async Architecture
41mLessons Learned Implementing a Resilient, Distributed, Real-Time Architecture
52mDistributed Computing Dealing with Time and Failure in the Wild
34mAntics, Drift, and Chaos
51mPatterns for building resilient software systems
53mResiliency and Availability Design Patterns for the Cloud