Using Statistical Techniques to Automatically Detect Game-Breaking Issues
summary
Netflix builds an hourly pipeline that collapses 20M+ distinct error lines into 135 keys and 26 clusters using regex pre-processing, sentence-transformers embeddings, and DBSCAN clustering, then ranks clusters by precision/recall/F1 to flag game-breaking logs. A real incident—game Oxenfree broke for a subset of users for three weeks tied to NGP profile response type name unknown—scores 83% precision, 67% recall, F1 ≈74. The system already surfaces issues (four incident investigations) and reduces noise for triage.
AI-written summary · about
chapters (8)
- 0:00Introduction to cloud gaming
- 1:21Unique delivery challenges
- 2:40Identifying broken sessions
- 4:07Selecting indicators
- 6:21Log processing and clustering
- 9:28Evaluating impact with metrics
- 11:52Incident analysis results
- 13:03Future work and takeaways
related talks
25mIncident Analysis Before Chaos Engineering
52mDevil in the Haystack
30mDebugging at Scale Using Elastic and Machine Learning
04mInterview
29mGaining Insights from a Black Box System
46mHow Cheap Learning (not deep learning) Finds Faults in Distributed Systems
34mIdentifying Root Cause via Playback Metrics
48mUsing Machine Learning to find Linux bugs