A Case Study of Natural Language Processing in Clojure
summary
Built a searchable Polish parliamentary corpus of ~300,000 speeches from 1991 to present and a pipeline that scrapes, normalizes and cross‑validates metadata, and lemmatizes with Morfologik plus a mapped national corpus. Uses skyscraper for structured, lazy, resumable scraping that handles hundreds of thousands of pages in tens of megabytes of RAM with file and MapDB caches. Produces a P4 corpus as a 265 MB corpus image and a desktop/browser concordancer Smirna (under 2,000 LOC) that supports metadata filters, frequency lists and keyword extraction.
AI-written summary · about
chapters (21)
- 0:00Start
- 2:08Why search corpora?
- 5:10My corpus: Polish Parliamentary Proceedings Processor (P4)
- 8:37Tools
- 9:47Skyscraper: overview
- 11:54Skyscraper: insights
- 13:45Skyscraper: abstractions
- 15:38Skyscraper: example
- 17:20Skyscraper: result
- 20:24Skyscraper: caching
- 25:18Smyrna: overview + Desktop app with a browser based UI
- 31:56English is easy
- 32:33Polish is hard
- 33:14Morphological analysis
- 34:20Morfologik
- 34:39Disambiguation
- 36:55On-disk representation: Corpus image
- 38:25On-disk representation: Indexes, etc
- 39:32Bitstream crunching
- 41:40Key takeaways
- 42:30Open source
related talks
28mCharting the English Language...in pure Clojure
33mBuilding Conversational Speech Annotation Tool in Clojure
44mData Science in Clojure
26mUnsupervised learning for news summarisation
41mUsing Clojure+Spark to Find All the Topics on the Interwebs
18mClojure Versus the Botnets
23mMultilingual Natural Language Processing
31mTaking AI/ML to the next level: Text!