Incremental Design, DevOps, Microservices & CICD
summary
Claims a working pipeline that auto-generates ETL from each microservice schema, writes Parquet to S3 and is migrating to Delta Lake, using change‑data‑capture for incremental loads. A custom Scala DSL breaks processing into many fine‑grained Spark Ops (they report ~300000 spark ops per day), with runtimes scheduled per op. Microservices remain independently deployable under continuous delivery, but teams hit coupling via published schemas, manual sunsets over Slack, and a platform effort (product managers, “planks” as stable API slices) to reduce developmental coupling.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:58Incremental design & architecture
- 10:15Independently deployable microservices
- 13:38Platforms
- 25:05The Beginning of Infinity
- 31:00Audience Q&A
- 38:20Where is the IT industry going?
- 43:40Outro
related talks
29mDevOps and Microservices Better Together
42mBetting On Evolutionary Architecture
45mResilient Predictive Data Pipelines
38mContinuous Delivery with a Microservices Architecture
36mWild Microservices and Imaginary DevOps
40mMicroservice Continuous Delivery
44mBreaking a Monolithic Devops
25mData-Driven Migration from Monoliths to Event-Driven Microservices