PanLex: Using PostgreSQL to implement a massive word-translation graph
summary
PanLex uses PostgreSQL to store ~25 million expressions from >2,500 resources across ~5,700 language varieties and yields 1.3 billion direct translations from those sources. The schema centralizes entries in a normalized denotation X table to avoid massive self-joins, and runs direct queries in single-digit to ~200 ms on a 128 GB RAM server with a ~50 GB core DB. Indirect translations use a geometric mean quality heuristic plus source grouping to rank inferred paths; the system and web translator work and scale for the current dataset.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:54What is PanLex
- 3:05Goals
- 3:59The goal
- 6:11Top 20 languages
- 7:15Schema concepts
- 10:29Meaning denotation concepts
- 12:12Meaning expressions
- 13:42Source
- 14:32Expressions
- 15:12Notation table
- 18:25Translation query
- 20:20Graph expressions
- 22:50Group score
- 23:52Results
- 24:41Irish Gaelic
- 27:29Geometric Mean
- 28:36Top Quality
- 29:28Intermediate Languages
- 30:42Performance
- 31:11Denotation
- 32:08Memory
- 32:37Speed
- 35:09Fake words
- 36:13Translation quality
- 37:21Indirect translation
- 39:50Translation interface
- 40:06Translation visualization
related talks
53mModern PostgreSQL for AI...
41mPostgreSQL is (probably) all you need
46mBig databases, many servers, data center or cloud - it's all here!
43mFull-Text-Search Explained from A to Z with French Food
33mPostgres, the Best Tool You're Already Using
50mFrom Transactions to Intelligence: Postgres for Open AI & Analyti
51mHow does PostgreSQL actually work
04mHow to play Wordle in Postgres