Improving the Usability of Remote Caching
summary
CI builds drop from 40–60 minutes to about 5 minutes after adding a remote cache; implemented working servers as a Docker NodeJS express + JSD (WebDAV) image and as an IBM/Apache HTTPD with mod_dav(+mod_dav_fs) config. Measured cache growth to 137 GB over ~10 builds, ~16 GB transferred per build, 80–90k HTTP messages per build, many zero-byte writes and lots of duplicate content writes; LRU eviction works but alternative policies (frequency, size, compute cost) could improve hit rate.
AI-written summary · about
chapters (13)
- 0:00Remote caching in Bazel
- 1:56TensorFlow build workflow
- 4:04Bazel remote cache options
- 6:39Remote cache with NodeJS
- 9:10Remote cache with IBM HTTP Server
- 11:21Bazel cache internals
- 15:58Cache storage challenges
- 16:20Cache hit rates and build times
- 19:10Cache traffic and network impact
- 20:37Optimizing cache writes
- 22:01Cache storage strategies
- 24:50Evaluating eviction policies
- 26:13Summary and future
related talks
41mSpeeding up Jenkins, Gradle, and Maven with a Build Cache
28mUniversal Cache: It’s time for fast, reliable, and cheap CI pipelines
23mWalk, Run, Fly - Take your build pipeline to the next level!
19mGetting to a green build with caching on CI
54mFrom Laptop Builds to Advanced CI
20mDevelopment efficiency with expert CI/CD strategies
50mBuilding on Key – Keeping your Actions and Remote Executions in Tune + Q&A
30mThe Bitter and the Sweet of Running a Planet-Scale Build &...