Software Heritage: DevOps challenges to preserve our software commons
summary
Software Heritage claims to archive 3 billion files and 700 million commits from over 50 million projects and exposes a Web API and planned git-clone mirrors. The system stores content once using a Merkle DAG deduplicated storage model, keeps two main copies (currently about 600 TB across two 300 TB arrays), uses PostgreSQL and Python3, and publishes code under GPL; operational pain points are transitive provenance queries, full‑text indexing at repository scale, loader/lister coverage, and long‑term object retention and replication.
AI-written summary · about
chapters (26)
- 0:00Start
- 0:59(Free) Software is everywhere
- 2:18Software source code is special
- 3:59Our Software Commons
- 4:51Software is spread all around
- 6:14Software is fragile
- 9:17Software lacks its own research infrastructure
- 10:27The Software Heritage Project
- 11:10Our principles
- 15:27Archiving goals
- 18:25Data flow
- 21:34Merkle trees
- 22:49Example: a Software Heritage revision
- 23:30The archive: a (giant) Merkle DAG
- 24:32Software stack
- 26:07Hardware stack
- 30:01A tour of the Web API - origins & visits
- 30:56A tour of the Web API - snapshots
- 31:28A tour of the Web API - revisions
- 31:48A tour of the Web API - contents
- 32:44Roadmap
- 35:11Challenges - scaling
- 38:36Challenges - operational transparency
- 40:32You can help!
- 42:12The Software Heritage community
- 43:27Conclusion
related talks
40mSoftware Heritage
53mIntroduction to Software Heritage, the Alexandria library for code
28mCodeCommons_ Towards transparent, richer and sustainable datasets for code generation model training
31mGitHub as a self-sustaining product for large-scale environments
14mGitHub's Data Story: Open Source at the Home for Open Source
57mOpen Source at commercial organisations
58mAdopting open source in your organization
25mHow to Grow & Mentor an Open Source Community & Achieve Collaboration with Over 170 Listed Corpor...