Building and Scaling Data Pipelines for Data Engineering With Ceph Object Storage
summary
Ceph RADOS Gateway sends S3 bucket notifications to Kafka, AMQ, or REST and triggers Knative-triggered serverless functions that scale to zero for realtime per-object pipelines. Demos include a pneumonia-risk X‑ray inference demo that tags or anonymizes images and a face/license-plate anonymizer; code and a Grafana dashboard sit on GitHub. Uses STS/IAM with role- and label-based access, archive Zone multi-site copies, s3a and NFS (Ganesha) access, and treats bucket inventory as a manifest to drive Airflow/ETL jobs.
AI-written summary · about
chapters (13)
- 0:00Introduction to talk
- 0:23Object storage in modern analytics
- 3:56Advanced capabilities of object storage
- 8:25Building data pipelines overview
- 9:07Event-driven pipeline with bucket notifications
- 11:39Serverless integration with KNative
- 13:25Medical imaging pipeline demo
- 17:59Anonymization pipeline example
- 19:43Common pipeline use cases
- 21:15Secure token services for access control
- 25:14Bucket inventory for batch processing
- 28:41Combining features for advanced pipelines
- 29:02Q&A session
related talks
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
35mStreaming and Real-Time Analytics
35mStreaming and Real-time Analytics
26mArchitecting real-time transactional data lake
21mOptimize and manage data on Amazon S3
22mTurn Data Into Insights
24mUncovering Community and Project Insights Through Data Driven Methods
31mOne Data Pipeline to Rule Them All