RGW S3: Feature Progress, Limitations & Testing
summary
Status update on RGW S3 compatibility, costs, and testing: RGW implements many S3 features but gaps exist (append-object, PubSub divergence, partial lifecycle and tagging support); deletions keep object heads synchronous, GC runs as single-threaded GC that slams OSDs and offers no progress metrics; lifecycle scans run per-rule causing up to hundreds of full-bucket scans. Proposals: store every object as a manifest and use a manifest layout for all objects, move metadata to a fast pool, add per-rule indexes, and expand s3-tests (pytest, Python3, Java/Go SDKs, v2/v4 toggles) and cross-provider runs.
AI-written summary · about
chapters (10)
- 0:00Introduction and terminology
- 1:24S3 and RGW feature evolution
- 4:51Understanding s3-tests
- 7:59Challenges in specification
- 10:31Object layout and performance
- 12:56Garbage collection mechanics
- 15:04Lifecycle processing issues
- 18:25Future testing roadmap
- 22:10Collaboration and ecosystem
- 24:32Q&A and concluding remarks
related talks
42mSIG Testing: Intro And Updates
38mSIG Cloud Provider Deep Dive: Testing Cloud Controller Managers
08mFour focus areas for Next-Gen Testing team
23m3 ways to test S3 in Python
34mAre You About to Break Prod? Acceptance Testing with Ephemeral Environments
24mgRPC Performance and Testing: A Maintainer Perspective
28mBig Data: From Testing to Governance
17mSSRD: Shapes and Summaries for Race Detection in Concurrent Data Structures