From Thundering Herd to Zero Outages: Building Reliable Inventory Sync
summary
A production inventory sync protocol that uses a 5-stage FSM (connection, app-init, sync-handshake, full sync, delta sync) runs on a tiered topology with a shared messaging layer (custom proxies → Envoy + gRPC) to reduce 100,000 client connections to ~1,000, handles 10,000+ servers with zero escalations over 4 years, enforces fairness via queued sync requests, batches 50 hosts (~20 minute worst-case drain) and drops eventual-consistency to ~5 minutes with an app-readiness check, all within 1 GB memory.
AI-written summary · about
chapters (15)
- 0:00Reliable inventory sync intro
- 1:46Challenge of 100k workloads
- 3:34Thundering herd of data
- 5:23Inventory sync protocol evolution
- 7:29Three-tier architecture
- 9:35Messaging layer for scalability
- 11:23First sync protocol failures
- 13:11Sync handshake for flow control
- 15:40Fair queue management
- 16:23Config parameters and batch sizing
- 19:17Final sync handshake protocol
- 21:46Application readiness five-stage
- 22:50Payoffs of five-stage sync
- 24:37Lessons from distributed systems
- 26:46Summary and Q&A
related talks
15mParaSync: Exploiting Fine-Grained Parallelism for Efficient File Synchronization
14mSyncWise: Error-Aware Time Synchronization for Reconfigurable Data Center Networks
28mMaking React Fun Again, with Sync Engines
22mRapid Development on Offline First Data Sync
14mScaling Large Production Clusters with Partitioned Synchronization
46mRethinking Syncing in iOS
24mEngineering Real-Time Data Synchronization Across Multi-Cloud Systems
30mScaling HDFS with Consistent Reads from Standby Replicas