Devil in the Haystack
Talkat AppSec California 2015 · on Tuesday, 2015-01-27· at 11:00· Annenberg Community Beach House: Track 2, 415 Pacific Coast Hwy·
summary
Claims a data‑driven, in‑app anomaly system that uses Apache Spark on Hadoop to profile millions of users with unsupervised online learning and sequence + distance detectors, turning actions into signals (geo shifts, concurrent IPs, rapid exports, out‑of‑order multi‑step workflows). Prototype runs as a backend detector, emphasizes rare‑event tradeoffs and feedback to cut false positives, and cites operational scale (billions of events) and an astronomy analogy with ~30 candidate alerts per day.
AI-written summary · about
chapters (9)
- 0:00Start
- 20:20Opportunities
- 23:21What is an Anomaly
- 24:24Heuristic Scenarios
- 25:58Anomaly/Outlier Detection Recap
- 31:35Behavior to Signal
- 36:43Apache Spark
- 37:34python spark RDD
- 38:27PySpark and MapReduce or Pig
related talks
19mBig Data
24mHunting Criminals with Hybrid Analytics
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mMachine Learning on big data in security applications
31mFinding Needles in a Growing Haystack
48mHunting Stealth Adversaries with Graphs & AI
01h 26mData Driven Action: a primer on Data Science
44mAnomaly Detection in Telecom with Spark