Keynote: Using the Python Data Science Stack to Determine
summary
Human Rights Data Analysis Group processes four source lists into a deduplicated roster using record linkage then plans a separate multiple systems estimation step to infer unrecorded deaths. Pipeline uses Python tools: pandas, numpy, scipy, scikit‑learn (feature vectors from Levenshtein Jaro and metaphone), XGBoost for classification, NetworkX plus hierarchical clustering and dendrograms for clusters. Starting 400,000 raw records yields about 44 million pairs to score; one problematic cluster hit ~12,000 and required splitting.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:07Organization mission
- 4:17Analytical methods overview
- 6:03Syria victim data project
- 10:43Record linkage problem
- 13:57Feature engineering
- 15:03Classification models
- 17:33Clustering challenges
- 21:29Hierarchical clustering
- 24:43Deduplication to estimation
- 25:48Ethical motivation
- 26:35Q&A
related talks
43mKeynote
37mData Science for Digital Humanities: Extracting meaning from Images and Text
27mData, Python and Humans
22mPython Applications in Infrastructure Planning and Civil Engineering
29m1 + 1 = 1 or Record Deduplication with Python
43mEveryone can do Data Science in Python
28mData Science for Social Good: Making Impact in Resource-Constrained Environments
32mEDA Techniques for Data Analysis