Building a multi-purpose platform for bulk data using sqlalchemy
summary
Claim: SQLAlchemy ORM is poor for bulk loads, so a domain config that resembles a model generates SQLAlchemy core statements and other artifacts. Implementation: CSVs are loaded into stage vs core tables, a reference-finding renderer fills foreign-key IDs, validations move only cleansed rows into core. Tools: SQLAlchemy core + generated SQL, CSV loader, SQLite demo. Result: generated SQL runs on the DB for high throughput and the demo populates and validates data successfully.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:31Let's Build... a Multi Domain Platform
- 4:20Data Model for Pub
- 5:37A CSV Delivery
- 6:34One Task: Find References Between Objects
- 10:00How To Implement Reference Finding? Core / ORM
- 12:22Next Customer: A Brewery
- 12:44Domain Model: The Lingo Package
- 14:32Task Specific Model Renderers
- 15:19Code Sample: Domain for Pub
- 15:53Code Sample: Find References
- 19:12Code Sample: Domain for Brewery
- 19:44Discussion of Domain Model and Task Specific Renderers
- 21:13Applications Of Domain Model
related talks
30mBuilding data flows with Celery and SQLAlchemy
32mNo More Raw SQL: SQLAlchemy & ORMs
30mThe magic of SQLAlchemy
01h 28mSQLAlchemy and You - Making SQL the Best Thing Since Sliced Bread
37mNo More Raw SQL with SQLAlchemy and ORMs
34mLet's talk Databases in Python: SQLAlchemy and Alembic
29mDatabase Sorcery 101 Ditch the Ancient SQL Scrolls for SQLModel
29mHow to build a data pipeline without data