DiSh: Dynamic Shell-Script Distribution
summary
DiSh is a shim that analyzes and transforms existing shell scripts into a distributed dataflow graph, then schedules pieces to cluster nodes while each node runs the original shell commands. On an HDFS example it cuts a 3.6 GB run from ~200 seconds to ~7 seconds by eliminating ~95% of network I/O; across benchmarks it reports about 13x speedup versus single-node Bash and ~2x better than Hadoop Streaming’s 7.2x. POSIX compatibility is maintained, differing from bash in only two test cases.
AI-written summary · about
chapters (5)
- 0:00Introduction to shell scripts
- 2:06Introducing Dish
- 4:27How Dish works
- 6:33Optimization process
- 8:25Performance and conclusion
related talks
13mFractal: Fault-Tolerant Shell-Script Distribution
22mPOSH: A Data-Aware Shell
19mDShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
25mStretch: A Fault-Driven DSM Runtime for Distributed Multithreaded Applications
43mNo shard left behind: Dynamic work rebalancing in Apache Beam
20mAdvancing HPC The Hybrid Scheduler Approach at the D.E.Shaw Group
16mHadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale
17mApache Nemo: A Framework for Building Distributed Dataflow Optimization Policies