Live Q&A Session #3

on YouTube ↗24m

" summary

Roundtable covers concrete practices: adopt Honeycomb-style observability with a three-tier oroboros deployment that uses dynamic sampling to avoid feedback amplification; enforce safety-first GNTP network rollouts with shadow tiers and 1% RC traffic; prefer fast reverts over risky forward fixes and automate rollback at fleet scale (10,000 vs 100 devices); treat stateful recovery as design-time RPO work or stream-and-replay live data for backfill; run weekly chaos tests (unplug a node) and codify reliability in reviews.

AI-written summary · about

" chapters (10)

1. 0:00Live Q&A intro
2. 0:48Monitoring to observability
3. 2:14Network ops safety culture
4. 4:22Teaching incident response
5. 6:29Volunteer release engineering
6. 7:56Recoverability when systems break
7. 10:05Debugging production tools
8. 14:22Network health reliability
9. 16:09Cross-service data consistency
10. 22:08Rollback procedures

" related talks

• 19mLive Q&AJoe Duffy, Sargun Dhillon, David Pariag, Joe Romano · @Scale: Systems & Reliability 2026
• 42mArchitecting for ReliabilityJason Yee · Dash 2022
• 46mPanel Discussion: The number of orcheBryan Hunt, Verocina Lopez, David Schainker, Thomas Depierre, Jani Leppanen, Jake Morrison · Code Mesh LDN 2020
• 22mRaise a Production System:Making on-call human-scale w/ Observability & TrustChris Sun · ScaleConf Colombia 2019
• 20mFireside Chat with Emily FreemanEmily Freeman · DevOpsDays Chicago 2020
• 52mPerformance & Stability Anti patternsAlex Mackey · NDC Sydney 2016
• 27mTradeoffs on the road to ObservabilityLiz Fong Jones · LeadDev Live 2020
• 31mTesting in ProductionAja Hammerly · RailsConf 2018