A Real-World Example of Using Network Behaviour to Detect Cyber-Compromise
summary
ANZ builds a Hadoop-based analytics pipeline ingesting NetFlow and DNS via Apache Flume and Kafka into HDFS, running Cloud Era Data Science Workbench, TensorFlow and third‑party models (PathScan / Apache Spot) to score network flows. The environment processes about 2 billion messages per day (1.5 PB stored, aiming for 10 billion) and surfaces anomalous lateral movement and staging patterns; data quality, reverse DNS, DHCP coverage and noisy non‑prod/VoIP sources require heavy cleaning and tuning, and a credential analytics beta correlates AD, Windows and proxy signals for prioritised SOC triage.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 3:09The Kill Chain
- 5:05PathScan
- 8:12Lateral Movement
- 9:15Source Data Issues
- 12:49Challenges
- 15:40Testing
- 17:43Advanced Cyber Analytics
- 22:00PatchySpot
- 22:53Advanced Cybernetics
- 24:43Operationalisation
- 28:28Data Quality
- 31:45Implementation Challenges
- 32:38Behavioral Analytics
- 37:05Referral Analytics
- 39:15What Should You Do
- 41:07Key Lessons
- 46:34Raw Data
related talks
25mThe Importance Of Real time Data In Fighting Cyber Attacks
01h 05mPractical network defense at scale
29mPractical ISP CSIRT Incident Handling w/ Network Flows, ELK Stack & Cybersecurity Intelligence...
36mCatch Me If You Can – Leveraging Cloud Foundry, SDN, BOSH and Predictive Analytics
03mApache Hadoop Meets Cybersecurity
27mCollect All the Data; Protect All the Things
49mRelational Observability for Cloud-Native Security and Data Science
19mBig Data