Designing Your Pipelines for Scalability and Resiliency
summary
Concrete guidance for enterprise scale: plan for 140,000 targets and multi‑thousand app on‑boarding (example: 6,000 apps with a 2‑hour deployment window), benchmark real workloads (chip customer runs 100,000 jobs/day), and design clustered orchestration with HA/DR, collocated artifact repos and replication policies. Standardize pipelines via DSL‑driven automated onboarding, parameterization and reusable modules, enforce early access controls and data‑retention/archiving, and measure key metrics with a single dashboard (StatsD/Graphite style) to find bottlenecks before they break the system.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:38What is Scale
- 2:37Plan Ahead
- 4:21Metrics
- 6:00Infrastructure
- 9:23Architecture Diagram
- 11:30Data Management
- 14:04Best Practices
- 18:55Monitoring
- 22:19Monitor Dashboard
- 23:05Audience Questions
related talks
55mPatterns & Anti Patterns for Scaling & Enabling DevOps Adoption
52mDevops @Scale
45mBecoming a Plumber: Building Deployment Pipelines
20mBuilding a Data Pipeline with Testing in Mind
45mResilient Predictive Data Pipelines
48mDevops@Scale
02mAdvanced Deployment Pipeline Techniques.
28mScale Your DevOps Initiative Beyond Its Awkward Teenage Years