Taming the 9s
summary
Claim: hit a product goal of Five 9s under 5 ms for an in‑memory ACID datastore but finds it hard to reach; work reduces long‑tail latency by applying continuous integration, targeted monitoring and profiling with Oracle Flight Recorder, Jenkins, Graphite and HdrHistogram, moving hot state off‑heap storage (mmap, DirectByteBuffer, Unsafe, JNI), GC tuning and watchdog/canary instrumentation. Result: worst 1s‑period spikes fall from hundreds of ms to tens of ms (typical ~25 ms, outliers ~80 ms).
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:23What are the 9s? Percentage of recorded events a given value
- 6:29Why long tail latency matters
- 7:24Developed VoltDB to expectations
- 10:57Continuous Integration as a starting point
- 13:29Outlier whack-a-mole
- 17:46Flight Recorder is the heat
- 21:08Old-gen GC
- 23:42Young-gen GC
- 28:38GC guided design
- 29:58Going off heap
- 31:19Unsafe safe memory
- 33:50So how about Linux?
- 35:16Enough with the environmentals
- 35:48Canaries
- 37:51Watch dogs
- 38:51Rate Limited Logger
- 39:34Debugging production
- 40:28Mesh Monitor
- 41:43How applications cause outliers
- 42:59Outlier tasks
