How to write well designed imports with Symfony
summary
Baseline import reads the IMDB TSV with 9.5 million records (~500 MB) via a generator-based reader, maps rows in a processor and persists in batches (250) with Doctrine EntityManager; full run takes about 1h03m and stays ~12 MB memory. Parallel “poor man” partitioning halves runtime; replacing partition manager with Symfony Messenger and queues (Doctrine/Redis/RabbitMQ) scales workers horizontally and hits hundreds of messages/sec. Built failure transports and retry (backoff 1s→2s→4s); solution works but adds concurrency, monitoring and operational complexity.
AI-written summary · about
chapters (12)
- 0:00Start
- 4:01Reusability
- 4:24Imdb Public Datasets
- 7:42Sequence Diagram
- 9:10Example Code
- 17:50Memory Usage
- 18:32Software Architecture
- 19:36Partition Manager
- 22:56Symphony Messenger
- 26:57Redis Queue
- 38:23Failure Transports
- 38:50Retry Strategy
related talks
50mSymfony Messenger: the sharpest tool in your PHP toolbox
45mMessy data != Messy code
57mEating spaghetti with Symfony
36mUsing migrations and fixtures
48mBuilding Really Fast Applications
52mRefactoring to Symfony Components
59mHow much database abstraction do I need?
34mSymfony and Elasticsearch: debugging and optimizations