Practical approach to building real-time analytics for cybersecurity applications
summary
Pipeline architecture and hands‑on choices for real‑time cybersecurity analytics using Citus + Postgres: a Kafka front end, per‑tenant sharding with a carefully chosen distribution key to avoid hotspots, native time partitioning to drop old data (3–5 days detailed retention), heavy use of unlogged tables and bulk COPY for ingestion, BRIN indexes and HyperLogLog for low‑cost distinct counts, Manual Query Propagation for aggregation tiers, autovacuum and maintenance_workers tuning, and a replication factor 2 deployment that supports fast rebalancing — used successfully at a Cloudflare pipeline.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 2:20Data pipeline for cybersecurity applications
- 4:48Attributes of cybersecurity applications
- 12:02Tuning Citus performance
- 17:42Features that make Citus stand out
related talks
25mThe Importance Of Real time Data In Fighting Cyber Attacks
47mBuilding a Real-Time Analytics Database
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
55mBreakout Session
32mBuilding Real-time Systems with Open Source Technologies
13mBuild Real-Time Analytics Platforms for Financial Crime Detection
20mLessons learned scaling our SaaS on Postgres to 8+ billion events
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns