Crashing BEAM applications

on YouTube ↗37m

" summary

Shows concrete pitfalls and fixes: converting untrusted strings with String.to_atom can exhaust the atom table and crash the VM; using links instead of monitors leaves orphaned processes; Cowboy keep‑alive plus per‑request Agents creates leaks under reuse, and a native DNS resolver pool of 4 causes queues that balloon to 100k processes during load; large binaries (>64 bytes) go to the shared heap and a long‑running message router for ~2,000 devices triggers OOM via retained refs. Uses observer_cli, Phoenix LiveDashboard, recon and Telemetry/Prometheus for diagnosis; fixes include monitors, temporary restarts, explicit DNS config, hibernation or forced full GC.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 0:12Crashing BEAM applications
3. 3:18Things we often hear
4. 3:33Actor Model!
5. 3:48Super scalable!
6. 3:59Well.. That's not the full picture
7. 4:58How to crash your BEAM application
8. 7:28Module names Node names • Struct fields "decode as atom"
9. 12:38For each request - Start a new Agent - Monitor the agent and the process - kill agent when process ends normally OR in case of exception: take breadcrumbs from agent, report, and kill
10. 26:43configures how often a fullsweep GC will happen, and that will collect ProcBins 2 hibernating process when it hibernates, a fullsweep GC will run 3 moving work to short-lived processes it possible
11. 28:58What to do when it happens
12. 29:12Make it operable!
13. 30:54Operate!
14. 33:32Metrics and Visibility!
15. 33:49telemetry
16. 34:45Bottomline?
17. 35:09There are several ways to break your systems
18. 35:37Don't go to production without visibility

" related talks

• 49mBEAM Concurrency in ActionSaša Jurić · YOW! Melbourne 2022
• 43mConcurrency Oriented Programming in a Modern WorldFrancesco Cesarini, Robert Virding · YOW! Melbourne 2022
• 51mThe Erlang VM or How I Started Loving Parallel ProgammingErik Stenman · GOTO Aarhus 2013
• 44mIdioms for Building Fault-tolerant Applications with ElixirJosé Valim · YOW! 2021
• 40mComparison and Commentary on "Application Operating System"C. Thomas Stover · Erlang Factory SF Bay Area 2014
• 52mConcurrency Oriented Programming in a Modern WorldFrancesco Cesarini, Robert Virding · GOTO Aarhus 2023
• 43mConcurrency Abstractions for Application SecurityBram Verburg · GOTO Aarhus 2023
• 30mScreaming Zombies and Other Tales: Race Condition WoesJoshua Larson · RubyConf 2020