Rethinking RDBMS Data Migration
summary
Claim: replace batch ETL with an event-driven pipeline that moves full records from RDBMS to an in‑memory store using Cloud Foundry, Spring Cloud Stream, and RabbitMQ/Kafka to achieve near real‑time updates and scale to millions of rows. Design uses a JDBC source + CDC to write event metadata, processors that join and emit full payloads, a Geode sink, error queues with automated 3‑attempt retries, synthetic events for initial loads, and PCF autoscaling — proven in production with sub‑second update goals.
AI-written summary · about
chapters (18)
- 0:00Introduction and speakers
- 1:03About Healthcare Service Corporation
- 2:29Data migration challenges overview
- 3:54Real-time ETL process design
- 7:26Moving to microservices architecture
- 9:57Spring Cloud Stream benefits
- 11:01Architecture and data flow
- 14:14Flexibility in data pipelines
- 16:19Event table and CDC approach
- 18:28Runtime management with PCF
- 21:16Demo setup overview
- 23:27Example data and event structure
- 25:58Initial data load simulation
- 27:22Handling deletions and deltas
- 28:06Performance and consistency
- 28:49Performance tuning with PCF
- 31:39Code simplicity and configuration
- 33:06Q&A session
related talks
32mRDBMS and Apache Geode Data Movement
39mModern Event-driven Architecture: Adopting Data in Motion
41mStreaming your database - Easier said than done?
37mDatabase Modernization at Cloud Scale: Minimizing Risk and Disruption
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
38mEfficient Reliable Database Migration: A Legacy Innovation Story
41mChange Data Streaming Patterns in Distributed Systems
18mData Modernization Using Confluent and Yugabyte