Best practices when accessing Big Data or any other data!
summary
Ellexus’ tools (Breeze, Mistral) trace real reads/writes and dependencies to find concrete fixes: one debug job from a single user crashes a shared filesystem after thousands of rogue writes; moving open/close out of a hot loop gives ~20x speedup; limiting small reads (<4K) raised a pipeline stage from 32h to 18h; AWS tests show GP2 SSD outperforms provisioned IOPS for bursty genomics I/O and realized >10% cost savings; changing MPI ranks so only I/O ranks open files cuts runtime ~10%.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:16Best practices when accessing Big Data ... or any other data!
- 1:03Ellexus Ltd: The I/O Profiling Company
- 3:12Ellexus enterprise products
- 11:57The noisy neighbour problem
- 26:24Housekeeping and deletes
- 27:32Failed I/O and file system trawis
- 33:15Opens and closes
- 44:26What is normal?
- 45:27Tuning cancer pipelines at the Sanger Institute
- 50:33Profiling the cancer pipeline
- 52:07Storage comparison
- 58:57How long did this work take?
related talks
40mPerformance Testing and Big Data
25mGreenOps at Scale: The 50TB Memory Diet for a Petabyte Platform
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
25mData Demons: Unleashing Chaos or Driving Performance?
44mAtomic Habits in writing high performing distributed code in the cloud
40mTop 10 data engineering mistakes #bbuzz
20mFull Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer...
48mDesign of data-intensive applications