Building a Recommendation Engine with Machine Learning Techniques
summary
Builds a SaaS Recommendation Engine using Java, Cassandra, Ruby on Rails, Clojure and Javascript and shows which pieces actually work and which fail. Uses RapidMiner and a KNN-based user/item KNN pipeline on pivoted utility matrices, tests with ~150 training examples and 18 test pages, and reports near‑perfect recall on some classes but failures where taxonomies overlap. Focus is on a content analyzer, a Wikipedia‑seeded feature set and the hard reality of feature engineering and a taxonomy API that still needs work.
AI-written summary · about
chapters (30)
- 0:00Start
- 0:30Building a Recommendation Engine
- 3:36What a Recommendation Engine Is
- 4:33The Paradox of Choice
- 5:29Illusion of Choice and Freedom
- 5:54Freedom and Commitment
- 7:04Second-Order Decisions
- 11:35Amazon
- 11:54Types of E-Commerce Environments
- 12:32Netflix
- 14:54Collaborative Filtering
- 15:36Content Based Approaches
- 19:28Learning Machine Learning
- 20:18Focus on Data
- 21:15Tensorflow
- 22:33Rapid Miner
- 24:24Sentiment Analysis
- 26:16Reviews on Amazon
- 29:23Utility Matrix
- 29:59User Based Recommender
- 32:46K Nearest Neighbors
- 38:35Performance Node
- 42:53Building a Recommender Component
- 44:18You Also Classify Users According to Their Content How Do You Get the Content for the Users You Have To Profile It There's a Lot of Different Ways To Do It Demographics for Example You Can Do Age Geographical Location Things of that Sort and Then Also if You Just Provide that Information in the Rate Item Similarly You Use Cholera to the Filtering To Figure Out What Their Tastes of those Other Users Are and Try To Test those in the Profile of the Unknown User so There's a Lot of Different Ways To Actually Mix the Two a Good Sample of a Content Based Recommender
- 46:05So Understand that Ax That Is the First Thing about Building Recommenders Data Cleansing It's an Arduous Job Sometimes I Spend More Time Cleaning the Data and Figure Out Where To Get the Data and How To Enrich the Data Then Actually Processing the Data in More Data It's Always Better Peter Norvig Which Is the Head of Ai at Google Right Now Will Tell You that Simple Models Will Trump Complex Models if You Have More Data so Whoever Has the Most Data Typically Wins that's Why Google Is Still Winning I Can at the at the Bottom of All this Their Specification Algorithms and I'M GonNa Also Now Move Again in the Machine Learning World There's a Lot of Different Algorithms so for Example When We Do Deep Learning We'Re Talking about Neural Networks Shallow Learning Used To Be like Two or Three You Guys Remember Perceptrons in in School You Know It Would Be like a Three Layer Neural
- 47:55In What I Did It's Use Wikipedia To Basically Create a Corpus of Knowledge about a Concept and Then Be Able To Grab a Specific Article in Front of Classified with that World Knowledge so the Idea Here Is that by Having Something like Wikipedia Which Is Curated and Updated Sometimes by Quote-Unquote Experts although I'Ve Updated Articles There and I'M no Expert but the Idea Is that Now You Have Really Good Training Data in Having that Fine-Grained Semantic Representation Allows You To Build Better Classifiers Also Wikipedia Provides a Taxonomy That's Built in When You Look at a Wikipedia Page You Have Basically the Parent Topic the Children Topics and You Can Really Use that Information To Create Features That Are More Relevant
- 48:50So for Example When You Process Takes You Tokenize and You Get Words out of the Text You You Remove Stop Words You Use Dictionaries To Clean It Up but You Still End Up with a Lot of Words That Do Not Provide any Value and Just Add to the Computational Burden of the Algorithms so One of the Things That I Did It's Basically Create a Crawler To Go to Wikipedia for Specific Topics Get that Information and Process those Documents To Create Vectors in this Hyper Space Now Once I Got the the Text out of those Wikipedia Articles I Did 150 Examples Just for this in 18 Test Pages so It's a Small Data Set When I Did It with More Data It Worked Better I Trained at the Simplest Classifier Which Is a Train a Cane in a Classifier
- 49:31I Trained at the Simplest Classifier Which Is a Train a Cane in a Classifier and I Classify a Collection of Webpages the Process Was Actually Fairly Simple in Rapid Miner in It Really Gave Me an Insight into How Things Work in Chute Work I Stored the Model So I Could Reuse It Later and Then What I Did Is I Store Multiple Versions of the Model and Then I Did a Gigantic Big for Loop To Basically Run the Different Models That Were Fine-Tuned Differently against the Same Data Programmatically
- 49:47I Stored the Model So I Could Reuse It Later and Then What I Did Is I Store Multiple Versions of the Model and Then I Did a Gigantic Big for Loop To Basically Run the Different Models That Were Fine-Tuned Differently against the Same Data Programmatically So I Could Really Get More Information about It Now the Data Set That I Use for the Example in the Presentation Came with an Amazing Rico 100 % Accuracy for this One Class So if an Article Was a Java Related Article in 99 4 % if It Was Machine Learning so once I Did this I Thought Man You Are a Data Scientist You Are Good at this You Drag and Drop Things You Connect Them and It Just Worked Amazingly Well I'M a Genius I Can Leave To Send My Rest of It To Google
- 51:32And that's How You Will Basically Get By in Machine Learning Otherwise You'Re GonNa Pull Your Hair and Scream at Your Computer I Did Not Do that That's Just a Web Search Type Bullet-Riddled Laptop So Again Overfitting Classifier Parameters Need To Be Tweaked Try Different Classifiers I Tried a Bunch of Them at this Small Data Set with the Training Data That I Provided They'Re Still Not Great so There's some Data Engineer That Needs To Happen Ahead of Time So Again Why Would You Build One of these Things It's Not Really a Product It's a Feature
related talks
23mFrom Data to Recommendations: Building an Intelligent System with ROR
01h 14mBuilding a Recommendation System with a Hybrid Search From Scratch
33mPeople who liked this talk also liked ... Building Recommendation...
43mBuilding your first recommender
39mA recommendation engine for your applications
08m“Test Recommendation Engine”
29mThe Challenges of Building a Personalized Recommendation System in E-Commerce
29mA recommendation engine in your PHP applications