Implementing a Text Classifier based on Lucene & LIBSVM
summary
Shows a working text-classification pipeline built on Lucene+LIBSVM that indexes training/test docs in Lucene, extracts bag-of-words features via posting lists, trains linear SVMs, and converts learned classifiers into rules as queries for very fast batch labeling of large indexed collections. For online use it builds a rules-as-index with payload boosts so one doc can be scored against thousands of topics (patent-scale). Systems run in production for newsletters, routing and product-location; heavy testing and GUI annotation improve quality.
AI-written summary · about
chapters (9)
- 0:00Introduction to text classification
- 3:38System workflow and classification steps
- 6:36Machine learning with LIBSVM
- 9:06Training data and expert annotations
- 12:28LIBSVM implementation details
- 15:07Vector space models in Lucene
- 17:25Batch classification challenges
- 20:34Customer use cases and results
- 21:54Q&A
related talks
31mCreate text classifiers in a few hours using the open-source Label Sleuth system
31mWhen humans teach machines
01h 24mMachine Learning with Text in scikit learn
01h 17mMachine Learning with Scikit-Learn
40mApache Lucene: From Text Indexing to Artificial Intelligence
46mPractical Machine Learning and Rails
34mClassifying Search Queries Without User Click Data
31mTaking AI/ML to the next level: Text!