Automatic Citation generation with Natural Language Processing
summary
Algorithmic citation recommendation for patents that uses topic modeling and a tf-idf / cosine similarity pipeline over 250,000 patents and abstracts, plus an implicit matrix factorization recommender on citation pairs. Implementation uses python, pyspark, Google compute, big query and cloud storage to parallelize parsing of XML and training; LDA experiments (200 topics, optimal ~40) run in ~1 hour, ALS factorization converges in <10 minutes on an 8-core cluster. Top-20 cosine picks contain ~10% of actual citations and overlap ~20% with the recommender; system aids humans, not fully automatic.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:05Patent citations matter
- 5:02Citation prediction approaches
- 9:32Human citation selection challenge
- 14:05Cosine similarity NLP
- 20:01Matrix factorization recommendations
- 27:30Q&A
- 34:19Technical performance
related talks
41mExploring Patent Data with Python
41mSimilarity Search in Patents via modern Artificial Intelligence
31mNLP text recommender system journey to automated training pipeline with Spark and Sagemaker
25mAnusua Trivedi, Data Scientist at Texas Advanced Computing Center (TACC), UT Austin
12mDeploying AI Literature Agents at Scale
31mCreating recommender systems with Spark, Scala and Prediction.Io
22mDeconstructing the US Patent Database
21mData-Driven NLP: From Statistics to GenAI