Storage and encryption in DuckDB
summary
DuckDB encrypts data at rest two ways: Parquet file encryption (since v0.10) and full database-file encryption that also covers WAL and temporary spill files. It encrypts at 256KB blocks inside ~120k-row row groups, compresses before encrypting, and uses AES modes GCM with 12-byte nonce (and CTR available) with a key-derivation step producing a 32-byte encryption key per database id. Block headers grow from 8 to 40 bytes (nonce+tag+checksum), WAL entries encrypt per-entry (length stays plaintext), keys live in a locked secure cache, OpenSSL acceleration is available via the HTTPS extension, and runtime/storage overhead is described as negligible though WAL growth can be noticeable under heavy updates.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:26Securing data at rest
- 2:30Parquet vs database encryption
- 3:13Encryption ciphers and AES modes
- 4:17GCM encryption and IVs
- 5:21Secure key handling
- 6:47Key cache and memory safety
- 7:31Storage structure overview
- 9:36DB encryption in 1.5
- 10:39Block-level encryption
- 11:24Block header changes
- 12:06WAL structure and encryption
- 13:51Encrypting WAL entries
- 14:34Temp file encryption
- 15:36Encryption API and OpenSSL
- 18:03Key takeaways
- 18:44Q&A: Block scanning
- 20:11Q&A: Random access and CTR
- 20:51Q&A: Key verification
- 21:34Q&A: Block header bytes
- 22:17Q&A: Nonce generation
- 22:58Q&A: Checksum necessity
- 23:19Q&A: Size increase
- 23:59Q&A: Key management
related talks
27mState of the Duck
28mAnalyze data easily with duckdb - and the implications on data architectures
40mDuckDB Meets Data Lakes
36mMotherDuck
30mChanging Data With Confidence using DuckDB
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
40mDuckDB beyond the notebook
40mDuckDB: Harnessing in-process analytics for data science and beyond