Speeding Up Data Ingestion in PostgreSQL: From Custom Methods to Multithreading and Atomicity
summary
Covers techniques to speed up data inserts into PostgreSQL, including batch INSERTs, custom insertion methods, and parallelizing the insert process. Explains how to maintain atomicity of the overall operation when performing concurrent inserts.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:00Basic data insertion
- 8:22Prepared Statements optimization
- 11:34COPY method for speed
- 15:29Multithreaded insertion
- 17:56Transaction atomicity issues
- 20:28Data updates and HOT optimization
- 26:20Two-phase commit in PostgreSQL
- 31:40Summary and method comparison
- 38:57Q&A
related talks
44mSpeeding up bulk inserts with Spring + PostgreSQL
52mBulk data processing and PostgreSQL thingy
30mIngesting A Million Time Series Per Second On A Single Instance
47mAnatomy of concurrent data access in PostgreSQL
25mData-intensive PostgreSQL: Three ways to scale
46mInvestigating Multithreaded PostgreSQL
41mHigh-Performance Ingestion with Python and Swarm64DB
32mLoading at Hyperscale: Advanced Data Import Techniques for Citus