Beyond KMeans - using LLMs to improve text clustering
summary
Combines sklearn KMeans and LangChain LLM calls to postprocess embeddings so clusters lose duplicates and “Misc” bins; uses OpenAI 3k+‑dim embeddings, runs KMeans (n_init tuned) then recursive KMeans to split large clusters below a threshold (default ≈ sqrt(N)), summarizes cluster samples with an LLM, merges similar summaries, and outputs final labels; tested on Safaria pages (poverty page had ~800 documents where vanilla KMeans suggested 3 clusters) and works as an offline Python package (llm_cluster_optimizer), not yet production‑optimized.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 1:12Text clustering goals
- 3:36Embeddings and K-means
- 7:19Optimization challenges
- 10:05The clustering bottleneck
- 11:36Proposed post-processing
- 14:03Refining clusters with LLMs
- 17:35Implementation and conclusion
related talks
49mLLM—Search Engine Killer?
41mHarnessing the power of LLM text embeddings
15mSearch Engineering Reimagined: Leveraging LLM in ML Workflows
39mCombining the Best of Two Worlds: From TF-IDF to Llama LLM
46mLLM—Search Engine Killer?
01h 01mWorkshop: Intro to topic modeling using Python
37mExpanding the Capabilities of LLMs Through Community Collaboration
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes