Herding Data to the Cloud
summary
Claims a working pattern to move multi-petabyte research data to the cloud that cuts on‑prem cost and unlocks scalable analytics: a federated data lake plus automated ingestion, normalization and REST APIs ingesting metadata via FHIR vocabularies, storing cold content in S3 and using starfish to profile files (4 PB on ~400 machines, 1,500 users, 600 TB–1 PB growth/year; top 50 users hold 80% of bytes). A research data platform feeds Lilly Insights for genomics and already runs as the first consumer.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:12HERDING DATA TO THE CLOUD
- 0:33Introduction
- 7:07Data Costs and Storage Consumption
- 10:55Making the New Data FAIR
- 15:49Handling Legacy Data
- 20:23RIDS Data Usage - Cumulative Distribution
- 21:10RIDS Data Usage - Access Time Heatmap
- 25:58Conclusion
related talks
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
37mCloud, security, and science converge in a data-driven world (PRT324)
17mIt's All in the Cloud
34mModern Data Platform, IoT & the Cloud -how...
01h 03mDiscovering the value of a cloud data platform
31mA Journey of Migrating Data Platform from Data Center to Cloud
07mCloud architecture for population-scale analyses of genomes and AI.
24mHow To Remove Data Siloes: Tips & Considerations