Design and Implementation of a Topic Detection System
summary
Cortico builds a production event detector that turns a few million tweets a day (and thousands of hours of talk radio) into short-lived “events” by constructing a word co-occurrence graph, pruning high‑degree words and single‑hit edges, and running Louvain community detection. Representative tweets come from cosine similarity to each cluster. It runs in 2–3 seconds per city per day, surfaces local stories used by journalists, handles Twitter well but struggles with noisy ASR talk radio, and produces mutually exclusive clusters.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:02What is Core Koza
- 3:15Earshot
- 4:25Geo Ranging
- 6:02Clustering
- 6:29Kmeans
- 7:30LD
- 9:52NMF
- 11:52Setting Up Data
- 12:21Card Index
- 17:17Intuition
- 19:22Modularity
- 21:55modularity optimization
- 22:20levain
- 23:42Prefiltering
- 26:31Final Algorithm
- 27:02Proof
- 27:20Output
- 28:15Limitation
- 29:21Retweets
- 30:19Example Hurricane Florence
- 31:40Cosine Similarity
- 32:42Clusters
- 38:59Reckless Idea
related talks
34mTopic modeling with BERTopic
41mUsing Clojure+Spark to Find All the Topics on the Interwebs
29mThinking of Topic Modeling as Search
45mDecoding Topics: A Comparative Analysis of Python’s Leading Topic Modeling Libraries Using Climate C
13mThinking of Topic Modeling as Search
30mCrash detector
31mWhen humans teach machines
28mNLP Solutions for infodemic management