To Batch Or Not To Batch
summary
Compares three implementations for importing a CSV into a database: a monolithic Spring Integration flow, a microservice pipeline with Spring Cloud Stream/Function and Rabbit, and a chunked Spring Batch job using JDBC writers and an H2/MySQL repo. Demonstrations process a 334-line CSV with chunk commits; integration and streaming produce duplicate rows on restart while Spring Batch uses a job repository and transactions to roll back to the last commit so restarts avoid duplicates. Notes orchestration and file-triggering remain external concerns.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:49Spring Batch
- 1:16Agenda
- 3:02Batch Processing
- 4:51Batch Jobs
- 8:13Chunk Based Steps
- 9:19Batch Architecture
- 10:25Batch to Start
- 17:58Data Integration with Files
- 24:28Microservice Cloud Streaming
- 25:38Spring Cloud
- 30:28SpringBatch
- 35:38Other Use Cases
related talks
50mWalking Through Spring Cloud Data Flow
39mRethinking RDBMS Data Migration
01h 06mOrchestrating Data Microservices with Spring Cloud Data Flow - Mark Pollack
42mRe-architecting Your Batch Processing with Domain-Driven Design
44mData Processing With Microservices
01h 01mCurrent Methods for Building Data Pipelines
52mStream Processing with the Spring Framework Like You’ve Never Seen It Before
48mSpring for Rabbit Super Stream and SQL filters