Lex Sleuther - A Novel Approach to Script Language Detection
summary
Tool called Lex Sleuther lexes files with six lexers and turns token counts into token-count vectors, then learns a mapping to six language scores via simple linear regression. It targets only six common script languages, is lightweight and open source (cargo install), and improves pipeline script detection so combined systems reach about 97% script efficacy for executable scripts at scale while remaining far cheaper and faster than heavyweight models.
AI-written summary · about
chapters (14)
- 0:00Introduction and speaker intro
- 2:27Problem with file type detection
- 5:58Existing tools and their flaws
- 7:26Introducing Magika
- 8:28Why build Lex Luthther
- 10:12Narrowing the scope
- 12:00Lexing instead of parsing
- 13:47How Lex Luthther works
- 16:54Training the model
- 17:36Results and CLI demo
- 18:40Performance evaluation
- 19:23Comparison with Magika
- 20:26Integration into pipeline
- 21:50Open sourcing Lex Luthther
related talks
22mEasier Lint Authoring with Semantic Search and (...)
14mLexy: A DSL Library For C++ Parsing
38mNext-Gen Detection: Harnessing LLMs for Sigma Rule Automation
29mLunarML: From Standard ML to Scripting Languages
26mCommand line language – where NLP and cyber security meets
01h 02mProgramming Language Research for Technical and Social Good: What PL Can Do for Good?
52mVisualizing Security via LANGSEC
47m Natural Language Processing & Anomaly detection in Sys call logs