Breaking Bug

on YouTube ↗36m

" summary

A subtle production bug causes long app hangs for ~3–6% of users: DNS returns an IPv6 address first and connections never complete until client timeouts, which froze the UI. Diagnosis uses Mixpanel funnels, Papertrail remote logs, proxies/Genymotion, Visor remote ADB to inspect real phones, and custom CDP network tracing. Fix includes adding MockWebServer tests and shipping a custom OkHttp DNS resolver (prefer IPv4) plus SSL proxies; it costs four weeks and ~15,000 lost users but cuts affected wait times ~50%.

AI-written summary · about

" chapters (18)

1. 0:00Introduction
2. 1:12Recyclerview bug horror story
3. 5:15Gathering user feedback
4. 7:19Suspects and failed replication
5. 10:15Remote logging setup
6. 12:28Simulating the bug
7. 13:52DNS typo facepalm
8. 14:59New performance issues
9. 16:46Remote debugging with Vysor
10. 18:11Systematic timeouts discovery
11. 19:18Measuring affected users
12. 22:32Network introspection tools
13. 24:21Designer confirms issue
14. 25:46Shared ISP investigation
15. 27:30DNS resolver fix
16. 28:12Impact and cost of bug
17. 29:40Scientific debugging method
18. 31:05Q&A session

" related talks

• 51mQ&A SessionAndré Staltz, Tal Kol, Ville Immonen, Eric Vlad Vicenti · React Native EU 2017
• 37mMobile Testing: Pinpointing Invisible ProblemsAlina Homenko · QA Fest 2018
• 23mMobile App Crashes in Production: Lessons Learned from a Tester’s ViewpointSrinivasan Sekar · HUSTEF 2024
• 23mTroubleshooting the fudge out of itCorina Adina Pip · HUSTEF 2019
• 45mUncovering breaking changes behind UI on mobile applicationsKazuaki Matsuo · AppiumConf 2019
• 31mUnleashing the Power of Remote ToolingEric Chiquillo · droidcon New York 2024
• 35m82 Bugs I Collected in a Year You Won’t Believe Made It to ProductionFrancois Martin · Voxxed Days Zürich 2025
• 51m82 Bugs I Collected in a Year You Won’t Believe Made It to ProductionFrancois Martin · NDC London 2025