ETL Confessions
summary
Benchmarks show CSV loads of a 1.1 GB (5.6 million-row) file finish in 17–36 seconds depending on tool and table settings; PG bulkload and psql COPY perform best, unlogged tables shave ~6 seconds, and piping compressed input to the server gives no consistent win. For one-pass queries, using a foreign data wrapper to treat the file as a table avoids forming unused columns and can be faster; for many repeated reads, COPY into a table plus aggregation or partitioned automatic tuple routing yields the best throughput.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:40ETL Confessions
- 1:08What is ETL
- 2:15Data Warehouse
- 3:13Extract Transform Load
- 6:08Load Tools
- 8:02Advantages
- 8:36Disadvantages
- 11:32Extract Load Transform
- 20:21Benchmarks
- 21:40Less IO
- 21:51PGLoader
- 22:33PG Bulk Load
- 23:10Regular Copies
- 25:04PG Loader
- 25:27Copy from Program
- 29:43Timings
- 31:43What I learned
- 40:30Which one will win
- 41:41Statement level triggers
- 42:28Thanks to Mote
- 42:58Questions
related talks
48mOptimal Data Loads
32mLoading at Hyperscale: Advanced Data Import Techniques for Citus
18mMaking Postgres inserts faster on Azure
41mUsing Databases to Pull Your Application Weight
25mPostgreSQL performance tips you have never seen before
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
59mFaster SSIS
30mIngesting A Million Time Series Per Second On A Single Instance