Building A Robust Cloud-Native Data Discovery And Delivery Platform
summary
A cloud-native MVP that indexes vendor datasets as per-dataset containers on Kubernetes, uses a metadata/query engine to extract fields, passes metadata through Knative serverless transformations, and writes searchable records into an Elasticsearch cluster. It enforces access with an authorization resolver and maps dataset names to active deployments via a proxy resolver, exposes client SDKs for authenticated fetches, and includes an approval workflow and indexing pipeline so teams find, authorize, and retrieve data without manual onboarding.
AI-written summary · about
chapters (16)
- 0:00Introduction and speaker intro
- 1:04Why data is the future
- 2:06Key data challenges in enterprises
- 3:30Data silos and advertisement issues
- 4:56Data authorization and contracts
- 6:40Cloud-native solution overview
- 9:04Data configuration and indexing
- 11:10Modules and query engine
- 12:15Onboarding data sets workflow
- 13:38Tech stack and transformations
- 15:27Data cataloging solution
- 16:09Data delivery problem
- 17:12Data delivery workflow
- 18:13Authorization and proxy resolver
- 19:59MVP and future enhancements
- 20:40Q&A and closing remarks
related talks
38mPlatform Building for Data Mesh - Show me how it is done!
27mRecipe for Building a Discoverable and Governed Data Platform.
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
33mBuilding Open-Source Cloud-Native Apps in JavaScript
01h 00mEventing with Knative and Cloud Run: From Basics to Advanced
35mCloud Native Data Platforms - The why, what and how
55mServerless On Your Own Terms using Knative
01h 03mDiscovering the value of a cloud data platform