Building scalable and robust Data Science Solutions
summary
Move notebooks out of production and refactor work into small, testable Python packages with clear APIs — aim for a sklearn-like API so users do fit/predict. Abstract the data layer, add input/output checks, logging and extensive unit/property tests. Package with setup.py, publish to a private registry such as jfrog artifactory or PyPI, and install via pip with --extra-index-url; building and deploying a package takes under 30 seconds. For scale, convert pandas pipelines to Spark (Databricks) and follow semantic versioning and release processes.
AI-written summary · about
chapters (4)
related talks
39mPyData in Production: Lesson Learned from Various PyData Deployment Strategies
05mProductionalizing Data Science Models
40mThe data scientists are gone. how do I deploy their work to prod?
38mState of Data Science & Machine Learning
34mData Products, From 0 to data science pipeline
28mThe Coding Portion of Data Science
46mProduction Model Deployment
35mGetting Machine Learning Models Ready For Production Using Python