Building a metadata ecosystem using the Hive Metastore
summary
Stitch Fix runs a standalone Hive Metastore (2.3x) backed by MySQL RDS and builds a surrounding ecosystem: a Bumblebee REST proxy plus Python client that returns table/partition objects as dicts and adds a tiny cache; a Yellowjacket thrift proxy that overloads get_table to hide test-data partitions and auto-generate “latest-partition” views; and a buzzcom event log published to Kafka by patched Spark/Hive for richer audit context. The system reduces metastore load, enables table cleanup and metadata-driven tests, and works but requires careful compatibility testing and migration.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:07About Stitchfix
- 2:04What is metadata
- 3:58The Hive Metastore
- 11:57Responsibilities
- 15:00Custom Patterns
- 16:59Hive Metastore
- 17:38Yellowjacket
- 19:47Test Data
- 23:40tracking the ecosystem
- 26:29learnings and future work
related talks
24mMetadata driven framework for data discovery and data quality
46mJourney of metadata
51mMetaConfig driven FeatureStore
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
34mApache Iceberg.
23mHadoop & Cloud Storage: Object Store Integration in Production
14mBuild your data lakehouse with Starburst Galaxy
49mData Platform Architecture, Evolution, and Philosophy