Building a Self-Service Marketplace for Test Data
summary
A small-scale test data management marketplace built from open-source components is presented: it uses an open-source visual pipeline tool (Streams on GitHub) that reads production snapshots, discovers related rows, samples data, applies masking (masked credit cards, preserved links) and runs JavaScript transforms. Processing uses a parallel Hadoop-style cluster for bulk jobs and exposes a REST/Swagger API to trigger pipelines; the demo shows previews, one-click extraction and masking, and claims cost savings compared to $2M/year commercial systems.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:55Test Data Management
- 6:56TDM processes
- 14:53Production data handling
- 22:38Demo: StreamsX
- 46:07Summary
- 47:59Q&A
related talks
46mBuilding a Self-Managing Test Data Marketplace
14mDesigning Secure, Compliant, & Scalable Test Data Pipelines for Enterprise
34mTest Data Management Principles
39mModern Data Pipelines Testing Techniques: A Visual Guide
42mDeciphering the way data is tested
25mTesting Data Pipelines
23mDDataflow: An open-source end to end testing from machine learning pipelines
20mAutomation beyond test - Test Data as Code