Generate test data quick, easy and lots of it with the Databricks Labs Data Generator
summary
Databricks Labs Data Generator uses Spark to create synthetic data at cluster scale, generating 100 million rows in 11s on a 12‑node (104‑core) cluster and scaling to 1B in ~3m, 10B in ~18m. It reads table schemas or specs, emits DataFrame output and writes to Delta or Parquet, supports regex and Spark SQL expressions to relate columns, and achieves repeatable referential-like joins via seeding. Requires tuning shuffle partitions to match cluster cores and some example integrations (regex, Faker) need manual fixes.
AI-written summary · about
chapters (11)
- 0:00Introduction and setup
- 3:18Need for synthetic test data
- 6:10Installing the data generator
- 8:57Generating 100 million rows
- 10:20Scaling to billions of rows
- 11:03Customizing column generation
- 13:10Using regex for data patterns
- 14:12Generating from existing tables
- 15:15Referential integrity in data
- 17:21Limitations and integrations
- 18:49Summary and Q&A
related talks
40mProductionizing Big Data - Stories from the Trenches
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs
49mBuild & deploy generative AI apps in days with Databricks Lakehouse AI
09mGenerating Synthetic Data with AI
36mBig Data Modeling with Spark SQL: Make data valuable
02mDatabricks at GoDataFest 2019
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
36mData Lakes with Azure Databricks