POSH: A Data-Aware Shell
summary
POSH is a drop‑in shell that offloads I/O‑heavy pipeline stages to proxy servers colocated with NFS storage, using an annotation language to declare which arguments are files and a greedy location assignment scheduler to minimize data movement. The system includes proxies on storage servers and a client runtime that preserves local semantics. Benchmarks show 1.6×–15× speedups: image processing ~50% faster in high‑latency settings, 80 GB log grep across five mounts gives 2–2.5× (cloud‑cloud) and ~10× (university→cloud), and git add/status runs 10–15× faster.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:20Shell Data Processing
- 1:06Running Shell Commands over Remote Data • Remote filesystems enable us to preserve our local environment
- 1:44Bash Commands over NFS
- 2:35Near Data Computing
- 2:55Can we bring data locality to the shell?
- 6:04Challenge & Contributions
- 7:10What would an annotation language show? Given an arbitrary command line, what information does Posh need to decide what to offload?
- 8:23Determining File Dependencies of a Command
- 10:35Shell Annotations
- 11:11System Overview
- 12:22Scheduling
- 12:40Automatic Parallelization
- 13:35Overview of Greedy Location Assignment
- 16:02Implementation & Evaluation
- 16:51Image Thumbnails with
- 19:45Git Commit Workflow
- 21:02Conclusion: shells should consider locality Contact
related talks
10mDiSh: Dynamic Shell-Script Distribution
24mPLOVER: Fast, Multi-core Scalable Virtual Machine Fault-tolerance
11mRemote execution with 100 to 100,000 CPUs
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
19mDShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
19mSuki: Choreographed Distributed Dataflow in Rust
16mShoal++: High Throughput DAG BFT Can Be Fast and Robust!
16mPreview: Cloud Computing