Scaling Embeddings in Real-Time
summary
An end-to-end pipeline that turns a live Hacker News feed into up-to-the-minute retrieval context, implemented with a Bytewax Python-native stream processor built on Timely Dataflow that polls every 15s and partitions work across workers. HTML is cleaned and chunked with unstructured.io, chunks are encoded with a Hugging Face transformer, and vectors are written to Redis using redis‑vector; the demo populates Redis vectors successfully and the connectors are swappable (Pinecone, Qdrant) so the pipeline runs in real time.
AI-written summary · about
chapters (9)
- 0:00Introduction to real-time embeddings
- 3:11Why stale data matters
- 5:43What are embeddings
- 7:51Building a real-time RAG pipeline
- 12:05Introducing Bite Wax
- 20:32Creating embeddings with Hugging Face
- 23:22Storing vectors in Redis
- 25:46Querying the vector database
- 26:50Q&A and closing
related talks
28mReal-Time Revolution: Kickstarting Your Journey in Streaming Data
35mUnderstanding the News around the World with Web Scraping and NLP at Scale
52m100% Big Data. 0% Hadoop. 0% Java.
41mAdding Generative AI to Real Time Streaming Pipelines
52mModern Problems Require Modern Solutions
21mPredicting Hacker News with Beam and TensorFlow
01h 47mRadanalytics.io streaming applications workshop
36mReal-Time ML Pipelines: Feature Chaining with Chronon