Dirty Data Science Machine Learning On Non Curated Data
summary
Claims that string‑aware encodings and simple pipelines let supervised models learn directly from messy tables without manual curation, demonstrating a dataset with 400 unique entries for 10,000 rows. Implements a dirty_cat similarity encoder and a target encoding for tree models and integrates them via ColumnTransformer + scikit‑learn, showing n‑gram similarity helps and that HistGradientBoostingClassifier with native missing‑value support or mean imputation plus indicators often outperforms costly iterative imputation. The dirty_cat library and experiments report improved predictive performance.
AI-written summary · about
chapters (5)
- 0:00Dirty data challenges
- 1:50Vectorizing pandas DataFrames
- 5:48Encoding categorical data
- 19:44Handling missing values
- 27:52Dirty data workflow
related talks
32mPreparing Messy Data for Supervised
42mHow am I going to deal with all of these cats
24mMachine learning techniques for data cleaning
31mHoly D@t*! How to Deal with Imperfect, Unclean Datasets
19mSkrub: Less data wrangling, more machine learning
30mData-Centric AI Cookbook-let's prep that data
24mThe art of categorical encoding for Tabular data problems
41mModels Schm-odels: Why you Should Care about Data-Centric AI