Maintaining Consistency in a Massively Parallel Environment
summary
Proprietary package manager Midas (MPM) stores metadata in Bigtable and package data in Colossus, using a modified torrent-like transport to fetch petabytes/day and handle millions of fetches/day. Configuration files push every 10 minutes without creating new packages if checksums match; ephemeral packages auto-delete after two days while binaries use durable retention (default three months). Labels map human names to immutable hash IDs, ACLs control who can move labels, and local decryption plus signatures enable rollback and tamper detection.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:29Solution: Google's Package Manager
- 3:02MPM Package Characteristics
- 4:36Case Study: Project Xyzzy
- 5:35Assumptions and Constraints
- 7:49Package Creation, cont.
- 8:58MPM Metadata
- 11:03Durability of Packages
- 13:18Package Distribution
- 19:40Metadata Distribution and Replication
- 23:55Security Access Control Lists
- 26:16Security - Encryption
- 31:23Re-visiting Project Xyzzy
- 35:25Labels
- 36:16filegroups
- 37:14Web Interface
related talks
30mConfiguration Management Patterns
29mHow packagecloud.io works
27mModelPack: An Open Standard for Packaging, Distributing and Running LLMs...
49mModern Software Delivery: Triumphs and Failures
30mThe state of Package Management on Kubernetes
40mBuilding a Modern Infrastructure Stack
01h 11mThe State of Package Management in C++
38mBigtable: A Journey from Binary to Service and the Lessons Learned along the Way