Analyzing Public Conversation using LDA and Topic Modeling, in Python
summary
Builds Choir OSINT pipeline that starts from 10 seed Twitter accounts and expands by a simple ranking on followers keywords and retweets, collects tweets hourly and deduplicates by tweet ID, runs preprocessing (regexes to remove RTs links emojis) and a daily analyzer. Uses Memphis stations 1/3/7-day retention as buffers, runs LDA (default shown as 5 topics × 10 words), pushes results to BigQuery for Tableau. Demo processes live feeds and produces readable topics, though dashboard refresh can lag.
AI-written summary · about
chapters (14)
- 0:00Introduction and speaker intro
- 1:50Why analyze social media
- 3:19Social network analysis basics
- 6:31Building football conversation network
- 9:21Ranking influencers for network expansion
- 11:14Data collection and architecture overview
- 13:46Processing pipeline with Memphis
- 16:16Code walkthrough and setup
- 19:10Demo walkthrough and visualization
- 25:16Improving the pipeline and future steps
- 27:45Visualization and metrics
- 28:48Live demo of data collection
- 30:39Live demo of topic modeling
- 32:04Q&A session
related talks
01h 01mWorkshop: Intro to topic modeling using Python
45mDecoding Topics: A Comparative Analysis of Python’s Leading Topic Modeling Libraries Using Climate C
01h 27mTips, Tricks and Topics in Text Analysis
01h 26mData Driven Action: a primer on Data Science
27mExploring Science on Twitter with IPython Notebook and Python Pandas
01h 51mNatural Language Processing in Python
49mWhy You Should Do Text Analysis in Python (Even if You Don't Want to)
44mAnalyzing social media data with Python