Scalable Feature Engineering with Hamilton
summary
Hamilton is an open source Python framework that turns each feature into a function so a DAG of dependencies is inferred and the driver handles execution, enabling unit tests, docstrings as docs, and runtime checks via decorators; it started at Stitch Fix, runs in production for years, manages 4000+ feature definitions, and speeds monthly model/feature updates 4x faster by decoupling logic from execution and delegating parallelism to Dask, Ray, Spark.
AI-written summary · about
chapters (13)
- 0:00Hamilton framework intro
- 1:08Feature engineering difficulties
- 3:15Time series forecasting challenge
- 6:45Solving spaghetti code
- 9:12Hamilton core concept
- 11:59Citrix production success
- 13:44Scaling feature engineering
- 16:48Distributed scaling frameworks
- 20:40Hamilton on Ray and Dask
- 23:12Pandas on Spark scaling
- 25:41Hamilton open source progress
- 26:23Hamilton future roadmap
- 28:29Q&A
related talks
24mHuman Friendly, Production Ready Data Science Stack
46mThe Python Data Ecosystem: Navigating a fragmented landscape
41mFeature management for your JavaScript applications
27mJosh Wills, Director of Data Science, Cloudera @ MLconf SEA
38mFantastic ML apps and how to build them
18mAccelerate development with the Engineering Development Hub
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
46mHarness Engineering: How to Build Software When Humans Steer, Agents Execute