The art of challenging assumptions

on YouTube ↗47m

" summary

WhatsApp runs 16,000 Erlang nodes and expands a distribution cluster to 3,000+ machines to test limits, finding CPU/network barely change but service publish latency breaks health. Tracing (recon_trace, redbug, sequential traces), benchmarks and automated tests pinpoint a slow registry and an API misuse: an insecure random twice as slow as the secure version. The team builds scalable process groups (overlay service discovery) and runs prototypes in shadow mode, proving it scales to thousands of nodes before cutover.

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 1:27Rise of over-the-top messaging
3. 3:47Based on actual events
4. 5:29Third time's a charm
5. 6:49Reality check
6. 7:28Identifying assumptions
7. 10:14Categorising assumptions
8. 12:11Evaluating assumptions
9. 12:32Erlang Distribution cluster
10. 13:45WhatsApp Meta-Cluster
11. 15:55Service publishing & discovery
12. 23:45Evidence collection
13. 30:51Benchmarking
14. 33:17Instruments: root cause analysis
15. 35:47Discoverability
16. 36:51Critical Thinking Practice
17. 37:517 flags
18. 41:52Post-completion error
19. 43:01Questions?

" related talks

• 33mHigh availability with Elixir and ErlangSaša Jurić · Full Stack Fest 2016
• 41mWhat NOT to do when building large scale systems in ErlangChandru Mullaparthi · Code Mesh 2015
• 42mThe world runs on bad softwareBrandon Keepers · LoneStarRuby Conf 2011
• 38mPerformance in the WildCraig Andera · Clojure/conj 2011
• 41mHopelessness and Confidence in Distributed Systems DesignCamille Fournier · Strange Loop 2015
• 57mMaximising Side Effects When Adopting Functional ProgrammingDan Macklin · GOTO Berlin 2015
• 01h 00mKeynote: Over a Century of ProgrammingJoe Armstrong, Robert Virding, Mike Williams · Erlang User Conference 2013
• 01h 01mArchitecture Decisions: A Case Study -Konstantin Kudryashov · DDD Europe 2020