Building Self-Service Data Platforms
summary
Claims a metadata-driven system that auto-generates Apache Airflow DAGs from declarative pipeline specs, ties into CI/CD for automated deploys and rollbacks, and uses PySpark for data quality checks plus Selenium for regression validation. Infrastructure relies on Terraform, Helm charts and Argo CD for GitOps, with monitoring in InfluxDB and Elasticsearch/Splunk and tests for regression, performance and backward compatibility. Says a team of about 12 runs a cloud platform handling production pipelines continuously.
AI-written summary · about
chapters (15)
- 0:00Introduction and Speaker Background
- 0:41Experience with ETL Tools and Team Management
- 1:35Current Projects and Cloud-Based Solutions
- 1:49Database Technologies and Automation
- 2:50Building Self-Service Data Platforms
- 3:02Platform Engineering and Infrastructure
- 4:55Modern Data Platform Challenges
- 8:20Metadata-Driven Pipeline Architecture
- 12:57Integration with CI/CD and DevOps
- 14:58Cloud-Native API Integration
- 17:01Apache Airflow and Python Integration
- 19:02Infrastructure as Code for ETL Development
- 20:11Data Quality and Testing Strategies
- 21:15Monitoring and Observability Standards
- 24:39Governance, Compliance, and Conclusion
related talks
26mDataOps: Accelerating Digital Transformation
11mScaling MLOps with Self-Service Platforms
35mOrchestrating success
17mDemocratising data analytics
37mBuilding a Data Platform from scratch
19mDemystifying Modern Data Pipeline Architecture
27mOvercoming Obstacles and Building Self-service Platforms for Application Teams
33mAutomating Data Analytics: Collect, Transform and Deploy - Lead With Change