Managing scientific simulations with Python with RQ (Redis Queue)
summary
Demonstrates using Redis and RQ to farm independent simulation tasks across 20–50 cores on 5–10 machines, claiming an easy afternoon setup with no broker configuration. Builds a job queue that workers consume, shows monitoring with RQ dashboard and rq info, and runs a Traveling Salesman demo: brute force up to 8 nodes is quick, 20 nodes is too slow, so random permutation via numpy gives useful approximations. Notes limitations: code must be distributed, arguments must be picklable, RQ forks a new Python per job and generally works but needs extra robustness for production.
AI-written summary · about
chapters (9)
- 0:00Introduction to redis q
- 1:04Scientific simulation basics
- 4:23Limitations of current tools
- 8:43Redis and redis q explained
- 12:14Live demo and setup
- 15:17Scaling with traveling salesman
- 18:14Monitoring and scaling
- 20:13Limitations and best practices
- 23:43Questions and answers
related talks
32mPython's Parallel Programming Possibilities - 4 levels of concurrency
24mDistributed Web Scraping in Python
19mOptimizing Data Retrieval with Python Celery
31mScaling Python to thousands of nodes with Ray
40mHomemade scientific clusters in Python
01h 34mPython for Distributed Systems
33mAnd now to something ELSE: Real Time Data Processing @ billiger.de
40mScaling Python - Bank Edition