How to avoid breaking other people's things
summary
A performance fix that cuts an endpoint’s latency in half triggers a surge of batched webhooks a few times a day and pushes database CPU to saturation; scaling down the webhook handlers or increasing DB capacity restores stability. Practical controls include documenting edge cases and limits, exposing timing constants, adding observability to detect integrator reliance, and releasing changes incrementally with percentage rollout or targeted acknowledged comms plus a fast kill switch for rollback.
AI-written summary · about
chapters (17)
- 0:00Avoiding breaking integrations
- 1:17Accidental breaking change example
- 3:24Software changes that break integrations
- 6:50Why breaking changes happen
- 8:36Integrator assumptions about APIs
- 10:00Danger of observed behavior
- 12:27Edge cases and undocumented features
- 15:13Ad-hoc integrator communication
- 16:16Naming pitfalls in integrations
- 17:39Batch processing changes
- 18:42Observability to mitigate breaks
- 20:04Empathy for integrators
- 21:29Scaling change communication
- 23:14Incremental breaking changes
- 24:37Rollback plans and incident response
- 24:57Balancing caution and delivery
- 25:39Conclusion
related talks
25mAvoiding Overload in Distributed Systems
34mDistributed Systems - What Can Go Wrong and How to Get It Right
01h 01mLittle things and big things: lessons for building large systems.
30mHow Event Driven Architectures Go Wrong & How to Fix Them
01h 02mTools for simplified large scale systems
26mGrowing Pain: Scaling to 10K Workflows per Week
40mPrincipals of System Integration
34mAntics, Drift, and Chaos