Named Entity Statistics Using Apache Pig
summary
Uses Wikipedia link counts to compute MLEs for surface-form→entity probabilities, implemented as Pig scripts that use Olevia Cazale’s Wikipedia loader to extract links, group by surface form, count pairs, join counts and normalize. Processes the German dump (~3.7 million pages) on a small dev cluster in <1 hour versus the prior pipeline’s >1 week, handles multiword names, and yields lightweight priors for downstream disambiguation. Also mentions PDFSpotlight and production readiness for updates.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:48Problem Statement
- 2:25Context
- 3:25Probabilities
- 4:10Methods
- 5:43Probability estimation
- 6:13Code examples
- 6:30Page links
- 7:12Grouping
- 7:37Maximum likelihood estimate
- 8:22Runtimes
- 9:28Outro
related talks
59mExploring Wikipedia with Apache Spark: A Live Coding Demo
26mBuilding an Entity-Resolved Knowledge Graph: Why and How
21mCase Law and ML on Spark
35mYou've got questions. We've got answers! #bbuzz
01h 03mMicrosoft Data Explorer for Excel: Data Discovery & Transformation
43mMaking AI transparent with RAG on your own data
27mScalable Data Pipelines - an ML Approach
21mMeeting complex data load and data preparation challenges