Faster Data Imports With Concurrent Transactional Subqueries
summary
Shows how Neo4j 5.21’s CALL { ... } IN CONCURRENT TRANSACTIONS runs batches on separate threads to cut a 10 million row import from ~110s to ~26s, and matches APOC periodic iterate’s single-machine speed (APOC ~1.5s, native ~1.8s on a 500k test). Explains concrete pitfalls: concurrent visibility is non deterministic so move dependent MATCHes outside subqueries, on error break and on error fail behave differently under concurrency, lock collisions occur and require retry logic or application-level retries.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:15Data import challenges
- 3:26Call-in transactions solution
- 5:37Adding concurrency
- 7:48Native concurrent call-in
- 9:12Concurrency semantics
- 11:38Error handling in concurrency
- 13:02Lock collisions and retries
- 14:48Future improvements
related talks
28mNeo4j for Beginners: Pros and Cons, Based on Social Networks
32mLoading at Hyperscale: Advanced Data Import Techniques for Citus
42mA Race To The Bottom: Database Transactions Undermining Your AppSec
29mTraversing from Zero to Production with Graphs
45mSpeeding Up Data Ingestion in PostgreSQL: From Custom Methods to Multithreading and Atomicity
23mDisrupting Customer Data Platforms: Neo4j vs. the Status Quo
47mAnatomy of concurrent data access in PostgreSQL
41mConcurrency, Scalability and Transactions -- Myths and Surprises