Improving Write Scalability of CernVM File System with Erlang OTP
summary
CernVM‑FS adds a new publication architecture that lets multiple release managers write concurrently by routing uploads through a storage gateway that accepts object packs, writes content‑addressed blobs to HTTP/S3 storage, rebuilds SQLite catalogs and hands out exclusive leases per subtree to avoid conflicts the gateway exposes an HTTP API implemented with Erlang/OTP (Cowboy handlers, Amnesia for session state) while heavy hashing and packing stays in C++ workers via port drivers and is already in the master branch.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:50ATLAS CMS HTB
- 2:57silicon tracker
- 3:30Camera sensors
- 4:42Computing infrastructure
- 5:40Computer architecture
- 6:25CernVM file system
- 9:15Main components
- 10:26Design
- 13:00Publishing
- 14:21Existing workflow
- 15:45Properties and constraints
- 17:06New architecture
- 21:14Architecture
- 22:59Good parts
- 25:15Things to improve
- 27:28Darker Graph Driver
- 28:43CernVM Workshop
- 29:31Questions
related talks
45mOpen Storage in the Enterprise with GlusterFS
08mRapid Design and Effective Operating of a General Purpose Object Storage at RWTH
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
36mStorage Wars
44mConcurrency + Distribution = Scalability + Availability(...)
49mInto Production
15mPuppet and containers: A multi-tenancy deployment
27mSolving inefficiency challenges at Datacenter scale