Demystifying File Similarity for Malware Detection
summary
Compares fuzzy hashes and ML on 800k labeled PE features from the EMBER dataset and AVclass labels from CrowdStrike, evaluating similarity with label homogeneity. ssdeep/SSD returns ~20% same-family neighbors among 100 nearest, XGBoost gives ~75/100, and deep neural embeddings reach ~94–95/100; XGBoost still outperforms deep nets for binary and 20-way family classification (~90% accuracy). TLSH is noted as promising but not tested here because raw files are unavailable.
AI-written summary · about
chapters (23)
- 0:00Introduction to talk
- 1:46Why file similarity matters
- 5:13Signature based detection
- 6:37Need for feature extraction
- 7:00Hamming and edit distance
- 9:09Jaccard similarity and MinHash
- 10:52Limitations of edit distance
- 11:13Fuzzy hashing overview
- 12:58SSDeep explained
- 14:23SD hash approach
- 15:45TLSH advantages
- 17:32True metric in TLSH
- 18:34Limitations of fuzzy hashing
- 19:19Learning based similarity
- 20:00XGBoost for classification
- 21:25Neural networks for similarity
- 24:55Experimental setup
- 26:43Evaluating classification models
- 28:55Binary classification results
- 29:37Multiclass classification results
- 30:22Measuring similarity performance
- 32:09Key conclusions
- 33:14Future work and Q&A
related talks
31mAn AI Approach to Malware Similarity Analysis: Mapping the Malware Genome With a Deep Neural Network
25mEnhancing Malware Code Similarity Detection through Vectorsearch and TLSH
28mAn Open Source Malware Classifier and Dataset
30mData Driven Security
26mHow to Apply the Machine Learning Appropriate Way for Your Security Operation
36mNinja Correlation of APT Binaries
22mA Scalable, Ensemble Approach for Building and Visualizing Deep Code-Sharing Networks
47mPrevalent Characteristics in Modern Malware