Release Management with Devpi

on YouTube ↗27m

" summary

Builds a company-wide devpi server that stores immutable wheel packages with unique versions derived from git via setuptools_scm, uploads per-OS indices and a generic index for pure-Python, and composes team and platform indices with inheritance so installs pick correct binaries. Runs a replica+master high-availability setup so installs keep working during maintenance. Reports about 10,000 packages, ~350 indices and ~18 TB downloaded, and the workflow is in production across Jenkins, desktops and services.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 0:33Anti-Example (yes, that happened...)
3. 1:45Our Approach (move towards Immutable Infrastructure)
4. 2:48Packaging Example (powered by setuptools_scm)
5. 4:07Devpi Package Server (from Holger Krekel et al.)
6. 5:34Release Workflow
7. 8:09Devpi by Example
8. 9:43Basic Index Layout
9. 10:27Aggregation Indices
10. 11:45Open Source Packages
11. 12:19Consumer Indices
12. 13:21Some Statistics
13. 14:21Host Setup
14. 16:06High Availability
15. 17:22Executive Summary
16. 18:31Some Lessons (learned the hard way)

" related talks

• 31mDevpi for teamsGraham Williamson · PyCon AU 2019
• 34mdevpi: Your One Stop Cheese ShopRichard Jones · PyCon Australia 2014
• 37mAutomated deployment of Python packages for developmentAndrew MacDonald · PyCon AU 2016
• 01h 21mSecure Python Packaging & Release Using Continuous DeploymentJussi Kukkonen, Martin Vrachev · SupplyChainSecurityCon at Open Source Summit Europe 2022
• 48mIntegrating upstream projects downstreamTomáš Tomeček · DevConf.CZ 2020
• 35mImproving CI/CD Process for Cloud Native Python Applications with PyPI CloudJaehyun Sim · CloudOpen @ Open Source Summit North America 2022
• 24mBuilding a Sustainable Python Package IndexDustin Ingram · PyCon AU 2019
• 40mPiku: git push deployments to your own serversChris McCormick · linux.conf.au 2020