Big Data Deduplication and data matching using Python

on YouTube ↗29m

" summary

ABS uses Python to deduplicate and link census records to other datasets, finding a concrete error: 250,000 missing people after improved matching. Pipeline uses name/address/DOB scoring, blocking by meshblock/postcode/age/initials to cut O(n^2), a custom disk-backed dict to exceed RAM, and an MPI shared memory handle to avoid copying large lookup tables. Critical heavy loops are reimplemented in C, checkpointing and micro‑optimisations make the system practical and successful.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 2:34ABS data projects
3. 5:52Census matching issues
4. 8:40The matching algorithm
5. 10:43Big data challenges
6. 13:47Python and memory
7. 16:58Optimization strategies
8. 20:17Multi-processing and tips
9. 24:05Q&A

" related talks

• 24mMachine learning techniques for data cleaningCathy Deng · PyData Chicago 2016
• 40mScaling Python - Bank EditionAndrew Fulton, Adam Lewis · PyData NYC 2022
• 22mPython Applications in Infrastructure Planning and Civil EngineeringBen Chu · PyCon AU 2019
• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017
• 44mBig Data with Python & HadoopMax Tepkeev · EuroPython 2015
• 26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...Itamar Turner Trauring · PyCon US 2020
• 37mAccelerating Python Analytics by In-Database ProcessingEdouard Fouché · PyData Berlin 2016
• 43mPython at City ScaleHunter Owens · PyData Los Angeles 2018