Every way we DDoSed our Cortex
summary
Cortex operators get concrete fixes: open-source cortex mixin JSON for dashboards and alerts, use Jaeger traces to pinpoint slow spans (example: single queries hitting BigTable for ~21s each and one demo query taking 90+ seconds), enforce SLOs (99.9% writes <1s, reads <2.5s), keep recent two days in cache to avoid storage hits, limit ingesters to ~1–1.5M series each, tier tenants (small/medium/big) and avoid naive autoscaling of stateful ingesters.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:57Cortex Architecture
- 2:40The Right Path
- 8:32Alerts
- 11:30How to get alerts
- 13:06Why we postmortem
- 13:56Read path
- 16:20Query latency
- 23:14Read path debugging
- 24:02DDoS limits
- 27:33Query limits
- 29:32Free frontend
- 33:07Summary
- 36:00Questions
related talks
29mThe osquery Agent. Performance Tuning @Scale
49mResponding in a Timely Manner
23m30 Minutes To Zero: Slashing Debug Time With WhatIsWrongWithThis?
27mObserving Failure in Reactive Systems
29mWho Watches the Watchers? From Closed Observability To...
25mThe Importance Of Real time Data In Fighting Cyber Attacks
38mCruising Ocean Threat With TheHive, Cortex & MISP Without Sinking
38mReady Steady Crash