Handshake's Approach to Content Tagging with vLLM and Anyscale
summary
Handshake implements vLLM on Anyscale and Ray to run in‑house LLM inference, keeping PII internal, and pairs that with Airflow + BigQuery batch jobs for initial tagging and a realtime Beam pipeline for streaming inference. A reusable Python tool ("push button llm evaluation") sends batched prompts to AnyScale or OpenAI and writes results to BigQuery and Elasticsearch for ranking. The deployed system suppresses resume-style posts and, notably, suppressed resume impressions same week, shifting creator behavior and improving feed relevance.
AI-written summary · about
chapters (10)
- 0:00Introduction to Handshake's feed
- 1:05Handshake product and challenges
- 4:14Cold start ML challenges in feed
- 6:21Unexpected user behavior with resumes
- 9:12Batch solution for resumé suppression
- 12:17Streaming solution for real-time tagging
- 13:41Impact of LLM tagging on user behavior
- 14:44Building Handshake's LLM platform
- 18:36Conclusion and future scaling
- 19:19Q&A session
related talks
39mAccelerating Open-Source RL and Agentic Inference with vLLM
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
20mSecure, private LLMs in your cloud with Anyscale Endpoints
52mAnatomy of an LLM
33mDesigning LLM Applications for Massive Datasets
43mAI Development Lifecycle: Learnings of What Changed with LLMs
22mPractical Strategies for Implementing Domain-Specific LLMs
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone