Batch select, coroutines and a million RPS
summary
The talk examines techniques for performing batch SELECTs against relational databases (PostgreSQL used in examples) to achieve large throughput gains similar to JDBC batch updates/inserts. It shows several batch-select approaches, measures the performance improvements, and demonstrates how Kotlin coroutines can let you write per-entity processing code while under the hood executing batched database operations; the same ideas apply to Java with async frameworks and other relational DBs.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 1:29Talk goals
- 3:14Why Kotlin and DSL
- 4:59Billions of selects problem
- 7:12Standard select approach
- 7:34JDBC performance without optimizations
- 10:00Batch select and its benefits
- 11:46Batch select issues in real projects
- 12:07JDBC query cache mechanism
- 15:42Query cache limitations
- 17:28Different query plan problems
- 18:11Select performance with parameter list
- 20:40preparedThreshold in JDBC
- 21:44Optimal SQL for batch select
- 22:27PostgreSQL unnest and generate_series
- 24:14Batch select optimization results
- 25:18Coroutine-based request batching
- 26:46Task distribution issues
- 28:12Optimal task wait strategy
- 29:14Live coroutine counter for optimization
- 30:59Loop switching: clients and rules
- 32:43Optimal coroutine count
- 33:48Coroutine context for batch management
- 35:57batchProcess method logic
- 36:41Final results and performance
- 37:26Conclusion
- 38:07Q&A: Open Source
- 38:28Q&A: R2DBC vs JDBC
- 39:16Q&A: RPS and CPU load
related talks
45mManaging parallelism using coroutines
41mHandling billions of events per day with Kotlin Coroutines
52mBulk data processing and PostgreSQL thingy- 41mHandling billions of events per day with Kotlin Coroutines
40mOptimize Your Database Operations with Asynchronous Processing
32m@Experimental Kotlin coroutines
49mKotlin Coroutines in Practice
42mThe Resurgence of SQL