Powering global-scale predictive intelligence using HPC on
summary
Maxar runs NOAA's FV3 GFS on AWS using Intel MPI, AVX-512 and EFA, assembling 156 c5n 18x instances (and tests up to 234) in a placement group to cut a 16-day forecast to ~41 minutes versus NOAA's ~100 minutes, a 50% compute-time reduction and 93% reduction in initialization time. Descartes Labs spins up ~1,200 c5 nodes to reach ~1.9 petaflops for geospatial processing, uses AVX vector intrinsics for fast object detection on billions of chips, and automates ephemeral clusters that build, run, and tear down.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:14Agenda
- 1:49How do we define HPC?
- 6:20Intel Xeon processor family powers HPC on AWS
- 7:41AWS offers broad choice of Intel instances for HPC
- 10:33Defining NWP
- 11:51Defining success of the project
- 13:28Automated FV3GFS workflow within AWS
- 15:50Compiling the FVZGFS system on AWS resources
- 18:01Cluster performance comparisons
- 19:53C5n and EFA: Performance improvements
- 22:09Results from the Cirrus Project
- 27:16My background
- 31:06The Descartes Labs team
- 31:52HPC powers the data refinery
- 33:44Creating a cluster with Python
- 35:08NAS paralel benchmarks
- 38:33LINPACK benchmark on vs 65
- 42:54Python driver
- 44:30using vector intrinsics
related talks
34mHPC on AWS: Innovation without infrastructure constraints
16mA novel Cloud High Performance Computing infrastructure for user
28mAdvancing the future of space in the cloud
42mBuilding a green supercomputer in the cloud
39mWhat would you do with 10^6 cores? High Performance Computing in the AWS Cloud
42mHow Intel & AWS solve some of the world’s most urgent issues
48mAccelerate science, research, and innovation with AWS
01h 13mEngineering the Cloud for Tomorrow: Performance, Speed and Scale for AI Workloads