Text categorization with Apache Lucene

on YouTube ↗12m

" summary

Uses an Apache Lucene index itself as the trained model to do text categorization with no external ML libraries, indexing BBC dataset 2,225 articles into five categories and querying with a MoreLikeThis query + term vectors while aggregating by average score. The demo shows correct labels for news articles (tech, business, sports, entertainment) with simple index settings; it performs poorly on very short texts and is best for longer unstructured articles rather than tweets or short snippets.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to text categorization
2. 1:15Dataset and indexing setup
3. 3:05Live categorization demo
4. 6:13How term vectors work
5. 8:21Vector space model explained
6. 9:23Further resources and live demo

" related talks

• 25mApache Lucene: From Text Indexing to Artificial IntelligenceLucian Precup · Devoxx France 2024
• 46mFull-Text Search Internals with ElasticsearchPhilipp Krenn · code.talks commerce 2018
• 39mEntity Linking at scale with LuceneEdoardo Tosca · Berlin Buzzwords 2022
• 42mBuilding a real-time news search engine #bbuzzRamkumar Aiyengar · Berlin Buzzwords 2016
• 55mFull-Text Search DemystifiedPhilipp Krenn · JavaDay Lviv 2016
• 01h 04mApache Lucene 8, Apache Solr and Elasticsearch – an OverviewUwe Schindler · JAX 2019
• 37mAlgorithms & data-structures that power Lucene & ElasticSearchAdrien Grand · Berlin Buzzwords 2015
• 19mVectorize Your Open Source Search EngineAtita Arora · Berlin Buzzwords 2023