Protecting the Web at a scale using consul and Elk
summary
Architecure uses ELK (Filebeat → Logstash → Elasticsearch → Kibana) plus Consul + consul-template + ipset to detect, score and distribute blacklists for brute‑force attacks. System ingests logs from 12,000+ servers hosting over 1M sites, flags malicious IPs (≈64M brute‑force events in a 24h sample, ≈120M malicious web hits/day), pushes blocks to ~2,000 Consul agents, and enforces bans locally (nginx reload cadence ≈60s, some detections within ~5s). Result: dramatic drop in failed logins and scalable real‑time mitigation across the fleet.
AI-written summary · about
chapters (11)
- 0:00Start
- 2:22What Is the Scale of the Problem
- 5:21Attack Detection
- 6:51Data Collection
- 19:39Distributed Locking
- 20:44Messaging Layer
- 22:21Security
- 27:20Local Checks
- 27:49Event Watches
- 29:50Console Replicate
- 46:14Console Backup
related talks
01h 05mPractical network defense at scale
47mReversing Engineering a Web Application - For Fun, Behavior & WAF Detection
01h 30mBehind the scenes of an ELK system
54mLessons and Lulz: The 5th Annual Black Hat USA NOC Report
52mKnock Knock: Understanding Who is Using Your Web Applications
50mTowards a Holistic Approach in Building Intelligence to Fight Crimeware
30mSpotting Threats with Elastic SIEM
38mThe Black Hat Europe Network Operations Center (NOC) Report