Towards a High Level Linter for Data Science
summary
Prototype static analyzer assigns abstract datatypes to pandas/numpy/matplotlib/scikit‑learn objects and checks library call signatures to flag high‑level data science code smells. It runs as an lra‑based static pass over notebook code sequentially (ignores interactivity), encodes ~200 library rules, and emits definite or possible warnings like using arithmetic mean on a ratio series or computing mean on a standardized series, and flags pandas in‑place calls producing non‑object values. Implementation and examples show the tool finds subtle, non‑exception errors.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:25Project motivation
- 3:31Data science challenges
- 5:18Code smells defined
- 7:04Code smell examples
- 11:21Abstract domain design
- 13:30Type rules and static analysis
- 16:20Handling pandas
- 18:29Prototype details
- 19:12Analyzer execution
- 22:06Future work
- 22:48Conclusion and Q&A
related talks
20mHidden Hacks in Linters for Better & More Secure Code
28mPreventing headaches with linters and automated checks
27mAn Overview of the Python Code Tool Landscape 2023
27mPython Linters at Scale
41mBetter Code for Data Science
39mSoftware Analytics with Data Science on Software Data
50mPractical Advanced Static Analysis
16mVisual statical analysis tool