Data DevOps is Real!
summary
PayPal builds a governed Data DevOps platform used by thousands of engineers, analysts and data scientists that unifies application and data pipelines, supports batch and streaming, and enforces data validation, rollback of both code and data, and visualization at every stage. Implementation uses Python, NoSQL, containerized operators, customized Airflow/Celery plus Control‑M and Informatica connectors, Jupyter, Kafka, TensorFlow, Splunk and SignalFx, PagerDuty/Slack alerts, blue‑green deploys and a microservice/mediator architecture — a production system that “works beautifully.” workflow-as-a-service data validation microservice
AI-written summary · about
chapters (23)
- 0:00Introduction to Data DevOps
- 1:25Why Data DevOps differs
- 3:55Foundational principles for data platforms
- 5:18Market landscape and tools
- 7:02Stakeholders and tooling
- 9:51Platform requirements and needs
- 10:11Legacy tool integration
- 11:14Integration and monolith phase
- 12:39Microservices architecture evolution
- 13:46Six key pillars of implementation
- 14:30Core operating tracks overview
- 15:32Version control for data assets
- 16:55Data validation categories
- 18:22Deployment and rollback strategies
- 19:26Containerization of operators
- 21:15Monitoring and quality checks
- 22:59Preemptive validation systems
- 23:19Application lifecycle stages
- 25:25Visualization and feedback loops
- 25:45Common challenges in Data DevOps
- 27:12Security in data pipelines
- 28:14Future of the platform
- 28:57Q&A session
