Pwrake : a Distributed Workflow Engine for e-Science
summary
Pwrake extends Rake to run real workflows across machines using SSH and the Gfarm distributed filesystem, replacing shell invocations with remote processes and assigning tasks to hosts with local data. Benchmarks show a workflow of 1040 tasks yields about 10x speedup over d-rake style execution, and a Montage image pipeline demonstrates ~20% additional gain from locality on Gfarm. The system defines dynamic tasks in Ruby and scales I/O by using per-node storage.
AI-written summary · about
chapters (29)
- 0:00Start
- 2:22Today's Topic
- 3:02Geographically Distributed Computer Resources
- 4:18Distributed Computing
- 4:57Using Multi-Core
- 5:58Parallel Programming models
- 6:44Parallelize Processes
- 7:37Graph representation
- 8:37Example of Scientific Workflow
- 9:48Workflow Graph of Montage
- 10:48Scientific Workflow Systems
- 11:15Language for Scientific Workflow
- 13:58Aspect of Scientific Workflow (2/3)
- 15:42Dynamic task definition in Makefile
- 16:53Define Tasks in Loop
- 18:31Parallelism in Rake
- 19:44Design of dRake
- 20:52Interim Summary
- 21:33Our Approach
- 22:30Pwrake features
- 23:18Remote process call: SSH
- 23:40Pwrake Parallelism
- 24:27Pwrake performance: Empty tasks
- 24:42Remote File Access
- 25:22File I/O is important for Data-intensive workflow
- 27:43Performance of Montage workflow
- 29:18Demo
- 31:41Future Plan
- 31:51Conclusion
related talks
37mRhebok, High Performance Rack Handler
23mRSpec: GitHub to Jenkins to Spot Instance Parallelization and Back Again
28mRactors in Ruby 4.0: Message Passing Without the Pain- 23mRSpec: GitHub to Jenkins to Spot Instance Parallelization and Back Again
43mPolyglot Paralellism: A Case Study in Using Erlang and Ruby at Rackspace
28mCome Discuss Ractors and Concurrency with 'The Dude'!
42mResource-Oriented Architecture With Waves
41mHow Puma Works