The Great Migration
summary
Migration extracts heavy background work into separate services: a sitemap generator, repository indexers and a sliced statistics pipeline that remove CPU and GC pressure from the main server. Uses Groovy/Grails, Ratpack for tiny services, Jenkins to run jobs, ElasticSearch for queries, Redis+Resque-style queues and GPars for parallelism. At scale (80k repos, ~250k users, sitemap >40MB, index run taking ~3 hours for 600k files producing 400MB) this cuts interference, allows throttling/killing jobs and speeds new index support to weeks. sitemap generator, Redis-backed queue workers, GPars parallel fetches
AI-written summary · about
chapters (5)
- 0:00Migration introduction
- 6:47Sitemap issues
- 18:03Repository indexing
- 29:50Statistics architecture
- 40:45Summary
related talks
34mKilling the Dinosaur: A migration story
35mMigration without Migraines: Automatic Migration at Scale
23mOn the road from Monolith to Micro
34mSustaining Git Performance Under Heavy Workloads
15mMajor migrations made easy
30mTackling Technical Debt at Adobe: A Story
01h 00mLarge-Scale Changes at Google: Lessons Learned From 5 Yrs of Mass Migrations
21mThe great SOA migration