Arkouda: Interactive Data Exploration Backed by Chapel
summary
Prototype arkouda lets Python notebooks issue massively parallel array operations via a Chapel server, using 0MQ for messaging and a NumPy-like API so users can import arkouda as np. The backend is ~8,500 lines of Chapel, the project is six months old with five-six daily users, and it runs from laptop to Cray. Tests create ranges up to 10^12 length (~1 TB) and notebooks target interactive speeds on 32 TB datasets while accepting some Chapel performance tradeoffs.
AI-written summary · about
chapters (10)
- 0:00Project motivation
- 1:26Python and data science
- 2:58Developing analytical tools
- 5:15Technical architecture
- 7:29Comparing alternative efforts
- 10:06System interaction model
- 12:07Programming language choices
- 14:01Implementation details
- 17:05Engineering challenges
- 19:06Future and conclusion
related talks
10mAkash Accelerate 2025: Nirmal Krishman from Chakra
41mUsing Kaskada to Do Real Time AI
46mData Exploration & Experimentation with Notebooks in Azure
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
22mAccelerating Analytics with Apache Arrow
06mA quack at building scalable data pipelines with DuckDB
33mUsing Kaskada to do Real Time Machine Learning
33mArrow in Flight - New Developments in Data Connectivity