Identifying Hidden Dependencies
summary
Honeycomb deliberately kills and restarts stateful services (Zookeeper, Kafka, homegrown storage) to find hidden dependencies: manual experiments reveal a leader‑election bug that pauses alert cron jobs for 5–10 minutes, a broken Chef gem that prevents Zookeeper from restarting, and auto-replacement logic with bugs. They enforce SLOs (ingest 99.99%, drop <0.01%), run restart one Zookeeper at a time, automate weekly restarts for Kafka and retriever nodes, and never let a node live longer than 12 months.
AI-written summary · about
chapters (29)
- 0:00Intro
- 1:01A dozen engineers build Honeycomb.
- 1:40by ingesting telemetry
- 1:55We deploy with confidence.
- 2:15Continuous delivery is an investment.
- 2:29Continuity of operations even more so.
- 2:40Stable platforms empower innovation.
- 2:52but stateful services can be scary.
- 3:36We need velocity and reliability.
- 4:01Identify potential areas of risk.
- 4:16Design experiments to probe risk.
- 4:27Prioritize addressing risks.
- 4:40Measuring reliability
- 5:13Service Level Objectives define success.
- 5:27SLOs are common language.
- 5:50We store incoming telemetry.
- 6:41Data persistence is tricky.
- 6:54Experiment using error budgets.
- 8:11Long-running processes.
- 8:32Data integrity and consistency.
- 8:46Single points of failure
- 9:39Restart one server & service at a time.
- 10:02Monitor for changes.
- 10:13Debug with observability.
- 11:01Test the telemetry too!
- 11:20Verify fixes by repeating.
- 12:00De-risk with automation.
- 14:28What's ahead for us?
- 16:28Go faster on stable infra. Manage risk and iterate.
related talks
34mIdentifying Hidden Dependencies
23mBuilding & Running a Scuba-Inspired Observability Tool
39m30 minutes of Failure
37mStop Hitting Yourself!
56mFast & Simple Deployments for Code & Infra
44mKubernetes & Habitat
42mHow Habitat can Boost your Chef Ecosystem
51mHearts of Darkness: A Spring DevOps Apocalypse