How to defend your sites from AI bots
summary
Nearly half of web requests are automated and in gaming reach about 60%, with specific examples: GPTBot causes 24% of Diaspora traffic and Read the Docs cuts daily bandwidth from 800 GB to 200 GB by blocking AI crawlers. Practical defenses described include user-agent plus IP verification using MaxMind/IPinfo, JA3 TLS fingerprint plus HTTP-header fingerprints for cross-IP identity, rate limits keyed to fingerprints or session IDs, proof-of-work puzzles, and emerging HTTP message signatures and Apple private access tokens; an open-source UA list and a security SDK are available.
AI-written summary · about
chapters (10)
- 0:00Introduction to bot threats
- 2:23AI bots making problem worse
- 6:58Types of AI crawlers explained
- 9:23User agent string limitations
- 10:27IP verification methods
- 12:53IP reputation and geo data
- 13:14CAPTCHAs and proof of work
- 15:38Emerging verification standards
- 17:03Fingerprinting clients
- 18:50Defense strategies summary
related talks
24mBots & Humans: Maintaining AI agent and automation in an AI-bot web
38mRise Of The Machines - How Automated Processes Overtook the Web
54mDetecting and managing bot activity more efficiently
49mAre you human or a damn bot?
11mHow to Navigate Emerging AI-Driven Security
31mStrategies for Identifying and Mitigating Bot Threats
43mArchitecture of an Anti-Bot Protection
15mProtect your apps from digital attacks w/ a modern defense strategy