How We Scaled GitLab For a 30k-employee Company
summary
An engineering team migrates GitLab’s repo and uploads from local FS to Alibaba OSS object storage, removes Grit and targets libgit2 by implementing libgit2 custom backends (odb/refdb and pack handling) and HTTP range reads, experiments with SSH elimination and LVS/ssh2http, adds multi-layer caching including an on-disk cache backend, and measures outcomes: clone becomes ~2x slower and some page loads go from ~50 ms to ~5 s without cache but overall performance proves acceptable for a 30,000-employee deployment and an AWS S3 port is planned.
AI-written summary · about
chapters (12)
- 0:00Start
- 8:07Linux Virtual Server (IP Virtual Server)
- 9:25Complications
- 16:42GitLab Sharding
- 18:18How to deal with FS?
- 26:01loose OSS store (write)
- 27:35Example
- 30:44git push (small data)
- 31:26Fixture
- 32:13git push (delta)
- 32:35git clone
- 32:51git fetch (delta)
related talks
29mScaling Across the Enterprise
27mB How GitLab scaled Git access with Go and what we gained from it
46mFrom 1,000 to 5,000 Users: Scaling GitLab Without (Too Much) Pain
24mA One Woman Show of Migrating an Entire R&D SCM From Bitbucket Cloud to a Self-Hosted GitLab
22mHow We Migrated Thousands of Repositories to SaaS With Developer Experience
20mMigrating from BitBucket to
21mDriving Cultural Change to Improve Software Development and CI/CD
26mRunning GitLab and GitLab CI in Kubernetes for the Enterprise