Things might go wrong in a data-intensive application
summary
A data platform hosts several petabytes and moves several terabytes a day and yet meets new workloads it never tested: optimistic locking breaks under thousands of 24/7 devices, so a hybrid adaptive locking scheme with local and global locks is rolled out and stabilises; an abandoned cleanup feature suddenly generates millions of requests and is reimplemented with a distributed job queue (rescue then gearman) to repair millions of files corrupted by a padding bug that had a 0.7 percent chance to slip past tests.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:59Data intensive application
- 2:58Introduction
- 3:38Case
- 4:55Game Architecture
- 6:05Incidents
- 6:28New customer
- 6:52Optimistic locking
- 7:50optimistic and pessimistic logging
- 8:58global log
- 9:18root cause
- 10:05abdul
- 11:11Scalability
- 11:38Security
- 14:27Fixing corrupted data
- 16:08Fixing USF9
- 18:25Losing User Data
- 19:53Reliability Trends
- 21:13Three Takeaways
- 25:49People
- 27:30Wrap up
related talks
28mData corrupting architectures we know and love
59mWhy Databases Cry At Night
56mBreakout Session
34mToo Big to Fail
48mUnderstanding and Reducing Application Contention
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
42mA Race To The Bottom: Database Transactions Undermining Your AppSec
57mDSS Keynote — Back to the Future: Prepare Your Data Infrastructure for Anything