An Open Source Malware Classifier and Dataset
summary
Provides an open dataset of 1.1M PE hashes (no binaries) with 900k chronological training records split evenly between benign malicious and unlabeled, seven JSONL files and a 1.6 GB tarball; releases code to compute ~2,000 features (byte histograms, byte-entropy, sections, imports/exports, patterns) using LIEF, a vectorization step, and a pre-trained LightGBM gradient boosted trees model that achieves 93% detection at 0.1% FPR on the held-out November–December 2017 test set.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:34Why Open Datasets
- 1:37Amnesty
- 2:28Security Locks Datasets
- 3:05Malware Classification
- 4:11Ember
- 4:46The Name
- 5:00The Dataset
- 5:56The Training Set
- 7:16The Data
- 8:06Two Types of Features
- 8:40Calculating Features
- 9:29Categories of Features
- 10:04Section Information
- 10:35Strings
- 11:04File Size
- 11:21Feature Vectorization
- 11:51Training a Model
- 12:56Scoring the Model
- 13:38Disclaimer
- 14:26Code Base
- 14:44Python Notebook
- 15:44Feature Engineering
- 16:18Semisupervised Learning
- 17:04Offensive Research
- 17:35Demo Time
- 21:36Download Data
- 22:06Packed Samples
- 25:21Metadata
related talks
40mDemystifying File Similarity for Malware Detection
47mPrevalent Characteristics in Modern Malware
30mData Driven Security
40mMalware Reverse Engineering with PE Tree—OSS Inspired by COVID
47mWorm Charming: Harvesting Malware Lures for Fun and Profit
44mApplied Machine Learning for Data Exfil and Other Fun Topics
48mCatch me if you can: Building a Web Malware Analyzer using Machine Learning
37mBuild Automated Malware Lab with CERT.pl Open-Source Software