Clojask: Inviting Data Scientists to Distributed Computing
summary
Clojask is a distributed dataframe that streams large datasets and keeps only the minimum rows in memory so it handles larger‑than‑memory work without manual tuning, using Onyx for parallelism and currently multithreading only not cluster deployment. The API mirrors R data.table and SQL-like API and allows runtime function shipping to workers so custom column maps need no serialization boilerplate. Benchmarks show Clojask beats Dask on many row/column operations but lags on groupby/aggregate/join for small files (1.8M, 3.6M) because it writes intermediates to disk.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:26Why Clojask
- 4:22Syntax comparison
- 6:40Distributed computing
- 7:35Data table
- 11:08Data flow
- 15:20Execution rules
- 17:26Technical decisions
- 19:06Benchmark comparison
- 21:33Outlook
related talks
43mDoing Data Science with Clojure
24mData infrastructure to build bigger with less
35m"Clojure for Data Science in the Real World"
25mDistributed data workflows: PySpark vs Dask
26mDistributed Computing in Clojure and Spark
37mOnyx: Distributed Computing for Clojure
37mClojure Remixed: Pyroclast Data Processing
36mMoving Faster: Why Intent Media Chose Cascalog