Ceph Notification at Scale: Notification V2, an Enterprise-Level Feature
summary
Ceph’s notification_v2 adds concrete fixes: topic ownership and permissions to prevent cross-tenant leakage, multisite syncing so a single topic/notification created once replicates across zones, and queue telemetry (radosgw admin topic stats + perf counters) so operators see pending entries. It moves topics/notifications to per-topic RADOS objects, makes queue writes asynchronous, adds TTL and max-retries for failed deliveries, and a rgw notification log subsystem; Bloomberg deploys this at scale across ~10 clusters (4 DCs, 2B requests/day) in production.
AI-written summary · about
chapters (15)
- 0:00Introduction to Notification V2
- 1:03Bloomberg’s storage infrastructure
- 3:12Ceph RGW clusters at scale
- 4:37Current bucket notification feature
- 9:15What makes V2 enterprise-level
- 9:56Ownership and permissioning model
- 12:43Multide syncing in V2
- 14:53Telemetry and insights added
- 18:04Topic bucket mapping
- 19:07New lock subsystem for notifications
- 20:29Control persistency in V2
- 23:19Performance improvements
- 26:09Migration and user account support
- 30:06Future work and open trackers
- 32:13Q&A session
related talks
39mCeph in Scientific Computing and Large Clusters BoF
34mOver a Billion Requests Served per Day: Ensuring Everyone Is Happy with Our Ceph Clusters’...
37mIntroducing Ceph Mgmt-gateway: Unified Access, OIDC-based SSO, and HA for Manageme...
36mCeph Security BoF
35mData Security and Storage Hardening in Rook and Ceph
24mSecurity Process Improvements in Ceph and Rook
38mDelivering Enterprise-Grade Object Storage: Bloomberg’s IAM Journey in a Multi-Tenant Arch...
18mData Security and Storage Hardening in Rook and Ceph