10 Billion Messages in a Day, How We Built the Infrastructure That Delivers
summary
61 datacenters, ~40,000 VMs and ~3,000 deployments per day support 10 billion messages a day across SMS, email, voice and chat; email peaks trigger a Frankfurt→AWS pattern that caches large attachments to S3 to avoid on‑prem bandwidth limits; an April 29 memory‑leak takes both redundant switches down and forces a migration from Hyper‑V to VMware plus staged failure drills; on‑prem Kubernetes evolves into info kubernetes service using VMware APIs; accidental regex VM deletion leads to delayed‑destroy safeguards; an agent called Stanley AI now talks to backend APIs to provision VMs and assist incident triage.
AI-written summary · about
chapters (7)
- 0:00Introduction to the talk
- 2:19First crisis in 2018
- 7:15Second crisis the outage
- 12:14Third crisis WhatsApp business
- 16:25Fourth crisis decommissioning Rome
- 23:36Ongoing crisis agentic AI
- 27:10Q&A and booth info
related talks
11mHow To Scale Infrastructure
44mEmail at Scale: How We Survived 800M Emails/Year (and DNS)
32m"From On-Prem to the Cloud and Back Again"
46mManaging a Billion Transactions with Tech, Tactics, and Teamwork
40mApache RocketMQ - Trillion Messaging in Practice
42mAspects of Continuous Deployment – A Multi-Dimensional View
44mUniSender Architecture: Growth from 100M to 1.5B Monthly Emails
43mMulti-tenant HBase Solutions at Facebook