Building a Self-Managing Test Data Marketplace
summary
A pipeline ingests production snapshots, discovers and flags sensitive fields, filters out EU/GDPR and minor data, and then masks or synthesizes values like credit cards and phone numbers. It performs depth/width sampling to reduce large bank workloads into small, repeatable test sets, implemented with a StreamSets‑style TDM engine, connectors and cron jobs, a visual pipeline editor, one‑command Docker deployment, and evidence logs for compliance.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:31Test data problems
- 6:41TDM goals and sources
- 12:02Production data handling
- 23:12TDM architecture
- 27:18StreamsX demo
- 39:09Summary & Q&A
related talks
51mBuilding a Self-Service Marketplace for Test Data
14mDesigning Secure, Compliant, & Scalable Test Data Pipelines for Enterprise
34mTest Data Management Principles
47mEfficiency up, failed releases down; compliant production data for your test and dev environments
39mModern Data Pipelines Testing Techniques: A Visual Guide
15mRobust Coverage with Right Test Data | Need, Challenges, & Benefits
25mTesting Data Pipelines
47mTest Data Key to Effective Test Coverage