Realtime DNS Analytics at Cloudflare with #ClickHouse

on YouTube ↗10m

" summary

Cloudflare ingests DNS logs from hundreds of POPs, handling over 100 billion queries per day for 6 million properties and spikes to millions of queries per second, and stores raw log entries in a ClickHouse cluster to power customer-facing and internal analytics. The pipeline uses Kafka and log-forwarders to cope with late-arriving data, keeps three months of raw rows (trillions of rows) and precomputes aggregation windows (1m/5m/1h) so ad-hoc SQL queries over high-cardinality fields complete in seconds.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 0:12What is Cloudflare
3. 0:47Cloudflares infrastructure
4. 1:13DNS analytics
5. 2:12Custom analytics
6. 2:41What Cloudflare wanted
7. 5:54Cast
8. 7:24Architecture
9. 8:00How Cloudflare got here

" related talks

• 17mUsing ClickHouse® Database to Power an Analytics and Customer Engagement PlatformPrafulla Gupta · OSA Con 2022
• 30mBuilding For FastDale McDiarmid, Ryadh Dahimene · Big Data LDN 2022
• 42mClickHouse: what is behind the fastest columnar databaseOlena Kutsenko · Berlin Buzzwords 2023
• 55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using KubernetesRobert Hodges, Kanthi Subramanian · Percona Live 2022
• 34mClickHouse® - What is Behind the Fastest Columnar Database?Olena Kutsenko · OSA Con 2022
• 28mColumnstore RevolutionPeter Zaitsev, Hannes Mühleisen, Alexy Milovidov · Percona Live Amsterdam 2026
• 27mFast DIY O11y with OpenSrc Analytics & VisualizationRobert Hodges · SREday San Francisco 2025 Q2
• 41mData warehouse and Kubernetes: Lessons from ClickHouse OperatorRobert Hodges · Data Con LA 2019