Hops in the Cloud
summary
HopsFS becomes a multi–data‑center, POSIX‑like filesystem with strong metadata consistency, storing small files in metadata on NVMe disks and pushing a consistent changelog (about 4% overhead) to Elasticsearch for sub‑second free‑text queries over millions of files. The platform (Hopsworks) runs hundreds of GPUs, ingests millions of images in seconds, and reaches about 1.6M ops/s on a Spotify workload in Google Cloud; feature store and TLS security are implemented, and block‑in‑S3 is planned to cut cost.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:26What is Hops
- 1:51Hops unique features
- 3:43What can you do on your platform
- 5:20Kubernetes
- 5:50What is S3
- 6:47Analogy from databases
- 7:27The first databases
- 7:46The relational model
- 8:24Data volumes
- 9:52POSIX
- 10:59Metadata
- 13:19File System
- 16:54NVME Discs
- 18:05Logical Clocks
- 19:03Machine Learning
- 19:56Feature Store
- 21:10Platform
- 23:16Platform Overview
- 24:27Pipeline Overview
- 26:33Pipeline Challenges
- 28:57Features Store
- 32:35Developers view
- 33:50Conclusion
related talks
23mHopsworks Feature Store after 4 years: Lessons learned and what's next
45mMulti-tenant Flink-as-a-Service on YARN
42mApache Hop
34mFrom Data Centers to Cloud Native - Dave Zolotusky & James Wen, Spotify
46mContaining an Elephant: How we moved Hadoop/HBase into Kubernetes and Public Cloud
50mApache Hudi : The Data Lake Platform
14mFalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...