How To Generate Test Data for Your Database Project
summary
Generates reproducible test rows with Faker + pandas pipeline, writes to MySQL/Postgres via pandas.DataFrame.to_sql and to MongoDB via insert_many, and ships a runnable repo and env files. Uses SQLAlchemy only for connections, then adds an auto‑increment/serial primary key with ALTER after bulk insert. Splits work across CPU cores minus one with multiprocessing (example: 60,000 rows → 15 cores × 4,000 each) and shows progress with tqdm.
AI-written summary · about
chapters (13)
- 0:00Test data generation intro
- 1:16Choosing data generation approach
- 3:51Project dependencies setup
- 6:44Faker for realistic data
- 9:12Multiprocessing for speed
- 12:22MySQL and PostgreSQL connection
- 14:27MongoDB connection
- 15:28Schema with pandas
- 17:34Multiprocessing pool
- 20:50Adding primary keys
- 22:39Running generation script
- 24:07Verifying data
- 24:27Resources and conclusion
related talks
29mHow to build a data pipeline without data
48mAuto-generating an API using PostgreSQL, Django, and Django REST Framework
23mBuilding a multi-purpose platform for bulk data using sqlalchemy
01h 10mReal World Iterative Database Development
34mTest Data Management Principles
23mDjango with PostgreSQL superpowers
17mOpen Source for Data Quality
28mHow to not pull your hair out while providing data to the business