Large Scale Recurring Imports in Drupal 8: Architecture and Implementation Tips
summary
Describes a working import pattern for Drupal 8 that parses external XML/CSV/JSON into queue items then processes small chunks to avoid cron timeouts, using Drupal 8 Queue API and optional Jenkins continuous integration to schedule and retry jobs. Recommends custom entities for long-term, high-volume retention and reports successful tests at about 500,000 records. Presents a three-step flow—gather, parse, save—with parsing libraries and chunked queue processing to keep site performance acceptable.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:19When to use custom imports
- 4:30Requirements Architecture
- 7:57Options
- 9:25Sources
- 10:27Fields
- 13:04Hosting
- 13:57Problems
- 17:57Triggers
- 18:31Custom Form
- 20:45Recap
- 26:41Gating
related talks
33mMigrating data to Drupal 8
17mA case study: Ditching the paper for Drupal
55mOn Moving Mountains: Lessons for Migrating Data and Content for your Next Drupal Project
54mScaling up and accelerating Drupal 8 with NoSQL
46mAdvanced migration techniques
15mAura Import: Anywhere, Anytime
39mHow We Scaled Catalog Ingestion to Hundreds of Thousands of Entities
31mCustom Drupal data migration: a Georgia GovHub story