High-Performance Ingestion with Python and Swarm64DB
summary
Hardware-accelerated PostgreSQL plugin Swarm64DB plus Python and async I/O aim to push real ingestion of ~40‑byte rows from synthetic ~900k/sec benchmarks into the millions. Using asyncpg/async IO, multiprocessing and an FPGA-accelerated plugin, measurements show peaks like 2.25M rows/s on earlier setups and up to ~24.5M rows/s with tuned config; code stays small (~90 lines), it works but the database and I/O remain the bottlenecks before the 30M/sec goal.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:54Big & Fast Data by Swarm64
- 5:19Some Big and Fast Data Use Cases
- 5:51Use-Case: Trading & Compliance
- 8:32So, what's the goal?
- 10:35Assumptions & Sequential Approach
- 12:26Multithreaded Ingestion
- 13:18Multiprocessed Ingestion
- 14:12Is this not going to work?
- 16:45Asynchronous Data Ingestion
- 17:28Results with Postgres 9.6
- 19:02Swarm64: System Architecture
- 21:09Processing Architecture
- 25:15Next level: aioprocessing & asyncpg
- 26:02Results with Swarm64DB 1.2
- 29:48End of Story
- 32:38Promising: asyncpg Benchmark
related talks
30mIngesting A Million Time Series Per Second On A Single Instance
48mBuilding Petabyte-Scale PostgreSQL Deployments
43mIngesting over 4 million rows a second on a single instance
50mScaling Postgres With Some Help from Redis
49mPostgreSQL - the database-sister of Python
28mCAPI/OpenCAPI Accelerated PostgreSQL
06mAchieve predictable performance
45mSpeeding Up Data Ingestion in PostgreSQL: From Custom Methods to Multithreading and Atomicity