Building OpenDNS Stats
summary
OpenDNS processes growth from 8 to 10→14 billion queries/day (about 162,000 req/sec) and logs every request, so the stats system uses a two-stage pipeline (stage one buckets by network, stage two aggregates/stores) with stage two written in C++ and MySQL backends. Design choices: SHA1 domain keys, per-user tables, mcached hot set (98% hit in 28 GB), rsync queues, controlled OOM shutdown, Percona InnoDB LRU data dictionary and ~2x throughput gains; system runs in production.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:38Loglines
- 0:58Graphs
- 1:57Pipeline
- 4:42Data Storage
- 5:35Memory
- 7:22False Start
- 8:18Logline
- 9:05Map
- 10:08Data Structures
- 11:42Reduced Process
- 13:34Stats Database
- 14:16Website
- 14:45Summary
related talks
10mRealtime DNS Analytics at Cloudflare with #ClickHouse
40mThe new reality of DDOS
47mScaling: from 0 to 25 million users
34mHow We Handle Over 1 Trillion HTTP Requests ...
01h 00mHyper Speed: When Big Data Blooms
01h 00mHyper Speed: When Big Data Blooms
18mHigh-Load Systems: Challenges in Social Network Development
48m10 Billion a Day,100 Milliseconds Per: Monitoring Real Time Bidding