Snakes on a Cloud: "Data science, meet DevOps"
summary
Engineers use Forecast a Python analysis engine built with Pandas and NumPy to pull data from Splunk and Graphite (about 1 TB of logs and 12 GB of metrics per day), run analyses in Jupyter on IBM Bluemix, and automate alerts. Forecast produces time-span detections (CPU >97% and latency >3s), runs a linear regression to predict disk exhaustion months ahead, scales across hundreds of clusters in seconds, and yields actionable correlations rather than dashboards of vague examples.
AI-written summary · about
chapters (10)
- 0:00Data science and DevOps intro
- 1:28Operations in software engineering
- 2:32Physics to software engineering
- 6:06DevOps culture and practices
- 10:01DevOps relies on data
- 14:13Building data-driven tool
- 20:02Predicting disk space exhaustion
- 25:00Correlating data sources
- 27:39CPU and latency correlations
- 34:05DevOps culture for data scientists
related talks
48mThe Use of Python in the Data World
39mProductionising Data Science at Scale
33mAccidental DataOps: How we matured our organization with data, and how you can too!
42mRobust Automated Forecasting in Python and R
43mTowards a full stack python monitoring+analysis framework
38mData Secrets from a Platform Engineer
35mLive Demo
05mCloud Foundry Monitoring & Alerting