Tall-and-skinny QR Factorizations in MapReduce
Talkat Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013) · 2013-10-21..10-24·
summary
Runs 8,192 nonlinear heat-transfer simulations produce about 4.5 TB and get stitched into a 5 billion x 64 matrix (~2.2 TB) whose SVD yields reduced-order models that evaluate about 100× faster with roughly two digits of accuracy. QR is computed in Hadoop using a communication-avoiding TSQR (local QRs, reduce-tree of R factors), avoiding MPI; practical timings: reading 200 GB takes ~253 s, writing ~1,000 s, R by the new algorithm ~500 s, full orthogonal Q ~3,000 s.
AI-written summary · about
chapters (9)
- 0:00Physics simulation overview
- 1:34Case study on heat transfer
- 4:10Matrix construction and SVD
- 6:19Advantages of model reduction
- 9:17Jet engine acoustic research
- 11:21Tall and skinny matrices
- 13:58MapReduce QR implementation
- 23:14Stabilizing QR algorithms
- 28:17Conclusion and references
related talks
47mMultiscale Analysis on and of Graphs
05mHadoop in 5 Minutes or Less
01h 14mDistributed and concurrent optimization for machine learning
21mMaja Kabiljo, Software Engineer, Facebook Inc.
10mMapReduce in 10 Minutes
32mThe intersection of Quantum Computing and Distributed Systems...
56mEfficient array (and data) processing by leveraging structure
42mWhen Hashes Met Wedges - A Distributed Algorithm for Finding High Similarity Vectors