Real-Time ML Pipelines: Feature Chaining with Chronon
summary
Chronon (Zipline) is an open‑source feature+embedding platform that unifies batch and stream computation to serve sub‑100ms features and embeddings for two‑tower search and ranking. It wires Kafka/Kinesis/PubSub into a Spark + Flink pipeline that writes to a KV store, reuses Spark partial aggregates to produce point‑in‑time correct backfills, and calls model endpoints (example: Gemini embedder) for on‑the‑fly embedding chaining. Production features reach tens of thousands at Airbnb, power 100% of Stripe’s charge‑path models, and use semantic hash‑based compute reuse for cheap, isolated agent experiments.
AI-written summary · about
chapters (11)
- 0:00Introduction to Chronon
- 1:10Background and origins
- 6:21What is Chronon
- 8:28Streaming feature challenges
- 11:16Streaming infrastructure
- 15:51Point-in-time backfills
- 18:35Training pipelines
- 23:03Model chaining
- 23:24Agentic experimentation
- 27:11Safety and efficiency
- 33:09Q&A
related talks
26mIntroducing Zipline: An Open Source Feature Engineering Platform
38mReal Time ML Pipelines in Multi-Tenant Environments
11mBuilding Real-Time Feature Pipelines
34mZipline - A Declarative Feature Engineering Library
45mBuilding a Feature Platform to Scale Machine Learning
31mLeveraging GraphQL for Continual Learning in Real-Time ML Systems
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
41mAdding Generative AI to Real Time Streaming Pipelines