Building a Feature Platform to Scale Machine Learning
summary
Go‑Jek builds a production feature platform that makes feature code single-source and replayable: feature transforms run on Apache Beam/Dataflow, training sets live in BigQuery for months of historic feature columns, and low‑latency serving uses BigTable + Redis (Cloud Memorystore) behind a serving API with cache, TTL and load balancing. Targets: sub‑10ms lookups, BigTable nodes ~10k R/W/sec, avoid 150k duplicate key hits via the access layer. Result: faster time‑to‑market, fewer pipeline rebuilds, and more time for modeling.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 5:31Why do we need a feature platform
- 10:04Component breakdown
- 11:24Dispatch Model
- 13:22Data Warehouse
- 15:58Feature Transformation
- 17:30Storage
- 20:13Requirements
- 21:27BigQuery
- 22:47Latency
- 23:46BigTable
- 25:27Access
- 26:31Serving API
- 27:28Discovery
- 30:01Service Generation
- 30:24Features
- 34:51Impact
- 35:33Question Sheet
related talks
27mFeast: feature store for Machine Learning
30mHow Feature Stores Enable Operational ML
28mBuilding Developer-Centric ML Inference Platforms
32mA feature Store for Machine Learning
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
17mCreating & operating ML models from event-based data using feature stor...
38mReal Time ML Pipelines in Multi-Tenant Environments
45mAnatomy of a production ML feature engineering platform