Parallel processing using python and gearman
summary
Processes load large NLP models so processes cannot be spawned per item, models reach 0.5 GB and Synthesio handles 20–40 million verbatims per day. Solution uses Python clients and workers talking via gearman job servers, passing JSON or pickle payloads, with workers initialized once and reused. Built monitoring that reports queue size and active workers and an automatic response to demand by dynamic worker scaling. System proves redundant across servers and lets non‑Python services submit jobs without changing existing crawlers.
AI-written summary · about
chapters (8)
- 0:00Parallel processing with Python and Gearman
- 1:12CESIO’s social media data needs
- 7:17Parallel processing for NLP
- 8:41Multiprocessing and RPC alternatives
- 12:35Why Gearman was chosen
- 16:29Language classification worker
- 22:54Monitoring and scaling Gearman workers
- 25:50Summary and hiring
related talks
17mR & Python playing nice, in Production
51mSpeeding up deployment of machine learning models
33mDealing with the versioning of production-ready models
30mEnabling Multi-Language Programming in Data Engineering Workflows
42mBuilding a hosting platform with Python
30mReliable Data Processing with Minimal Toil
42mImproving Machine Learning Development Reliability
36mUnleashing Generative AI: Improving Developer Productivity in SRE