Using Embedding Layers to Manage High Cardinality Categorical Data
summary
A production system turns sparse search queries into usable features by mapping 10 million keywords into 50 dimensional vectors using pretrained GloVe where possible and a trained bigram model for the tail then loading those into PyTorch embedding layers that are frozen then fine tuned The embedding approach scales where mixed effects models fail to converge and it yields reasonable bid and revenue per click predictions versus GBMs while improving explainability through clusterable semantic behavior
AI-written summary · about
chapters (10)
- 0:00Modeling search term revenue
- 3:50Ad tech metrics and revenue
- 6:32Handling high cardinality
- 9:59Mixed effects models first
- 14:24Limitations of mixed effects
- 16:31Embedding layers solve it
- 20:43Implementing embedding layers
- 22:05Model trade-offs
- 27:20Aligning model with business
- 30:17Q&A
related talks
27mData with Purpose: The Power of Embeddings
38mEmbeddings! Embeddings everywhere!
45mBehind the hype: managing billion-scale embeddings in Elasticsearch and OpenSearch
13mTraining Embedding Model Resiliently for Multimodal M...
01h 08mGoing multi-modal
41mFrom Embeddings To Index: A Practitioner's Guide To Domain-Adapted Neural Search
42mHow am I going to deal with all of these cats
30mExtracting insights from unstructured data