Moving the Unmovable: migrating from VMs to K8s
summary
Red Hat CoreOS builds move from VM-based Jenkins to OpenShift/Kubernetes over three iterations in about six months, trading persistent Jenkins state for ephemeral Jenkins pods and a declarative job spec model. The pipeline uses CoreOS assembler to produce OS disk images in under an hour, runs ~20 developer pipelines plus four production snapshots, externalizes state to S3 and secrets/configmaps, and solves KVM device needs by temporarily granting then dropping privileges for KVM access via privileged pod.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:05Two Migrations: only one success Before coming to Red Hat, I participated in a failed Kubernetes Migration At $PREVIOUS_JOB, we were able to deploy, but we could never "rust" it. The workload never shifted. Only towards the end of the Pipeline did the lessons start to distill
- 5:13Do NOT feed after midnight: A six month journey • The biggest problem was our thinking "the pipeline" instead of 'n pipeline." We conflated the process with output Our requirement of moving juxtaposed to the K&S idea of "deploying
- 7:40The service's environment is the single largest obstacle to moving a service.
- 9:53If you think in terms of "moving," you are off to a bad start "Lift and Shift" almost never works. We moved the pipeline.
- 10:46Options are great: deployments are cheap • Resist the temptation to maintain feature parity, especially for organically grown services Declarative code means declarative behavior. Locate where the defaults come from.
- 14:22Use a Single Templating System OpenShift supports templating • We had four sets of defaults: o Makefiles The YAML templates The Jenkins Groovy o OpenShift build en Vars which set Jenkins parameters
- 17:03If you can describe the service in YAML/JSON, then you can describe the environment and config the same way. Embrace the YAML/JSON early.
- 17:34The Better Ways() • Option 1: config/secret map and "source" it • Option 2: teach your Gremlins to understand the YAML. We use both options. This gave us flexibility: YAML allows for us to describe N variants of pipeline Sourcing allows us to store config in the KS way, • ConfigMaps/Secrets are shareable within the namespace
- 18:54Environments: It must be loosely coupled Define the who, what and why of the environment • Place the environment setters outside the service itself. Git branches for Configuration and Code
- 20:53Jenkins: Not Cloud Native • We spent way more time in Jenkins land that we cared to • Custom Master and Slave Docker Images Scripted the Jenkins Setup Memory issues (Jenkins is greedy) • Had to implement CI for our Jenkins Pipeline. • Logs were painful Better than a beer after a long day: "oc delete pod/jenkins..."
- 26:32Namespaces, DNS, etc. For deployability, use variables when you can. If you rely on external resources, consider
- 27:24The second largest obstacle is how you develop, test and deploy services. {Dev Git}Ops anyone?
- 28:25Triggered by Jenkins and Jenkins being Triggered • A horrible mistake was trying to support the Jenkins UI and OpenShift build parameters. Truthiness. Pick on way for the service to be triggered
- 30:30Complexity Management At the number of pipelines exploded, so did the complexity. • When we realized that we could have all the knobs, we added them • The knobs were incredibly powerful Knobs are brittle • Clever code is dangerous code. • A lot of the knobs have been removed. • Carefully consider the cost of introducing complexity
- 33:19Six Months with Three Versions of OpenShift We started on UpShift, OpenShift 37 • Moved to a VM-based 3.11 Cluster Finally settled on a bare-metal 3.11 cluster with KVM access. One thing we did well was to plan for frequent deployment. From 3.7 to 3.11 took roughly two weeks • From 311 VM to 3.11 bare metal took 45 Minutes
- 34:30If we did it again • It would have been less work to start from scratch. • Challenge the assumptions and the dependencies Use OpenShift BuildConfigs natively. Break the monolith first. Composable pieces. Simplify. • Do not conflate the workload with the environment Determine what is really being moved. Better integration in the full OpenShift end-to-end tests
- 35:32Our thinking changed Service is now completely re-deployable We plan around Jenkins failing State is stored elsewhere (S3, Koji/Brew). • OpenShift is the environment, Jenkins is the execution runner Trained people to NOT look at Jenkins (that's for developers)
- 36:12Had we thought more about results and less about the "how" we would have made different decisions.
related talks
05mJenkins in the Cloud
25mContainer pipeline for devs and enterprises alike!
47mContinuous Delivery Meets OpenShift
23mOrchestration and CI/CD
05mMoving to Kubernetes, or not?
27mNext generation CI - building Eclipse projects on the new CBI build farm
17mOur Journey to Kubernetes at Scale
35mOff the Beaten Path: An Explorer’s Guide to Kubernetes