RDBMS and Apache Geode Data Movement

on YouTube ↗32m

" summary

Describes a cloud‑native ETL pipeline that moves relational data into Apache Geode using Spring Cloud Stream with RabbitMQ/Kafka and JDBC readers, splitting work into source/processor/sink microservices. Claims handle huge initial loads by creating synthetic events, processes 60 million records in about six hours, and sustains ~600,000 events per day with 15–20 ms API latency. Shows code, DB trigger/CDC options, batching to Geode, and a dead‑letter retry pattern that proves reliable.

AI-written summary · about

" chapters (15)

1. 0:00Introduction
2. 0:20Shoutouts
3. 0:48About HCSC
4. 1:31The Problem
5. 2:54Simple Extract Transform Load
6. 3:55EventDriven Process
7. 5:32Microservice Architecture
8. 7:17Spring Cloud Stream
9. 9:17Demo
10. 10:01App Setup
11. 11:55Creating Regions
12. 23:59Multiple Streams
13. 26:10Failed Messages
14. 30:21Metrics
15. 31:18Outro

" related talks

• 39mRethinking RDBMS Data MigrationAnupama Pradhan, Jeff Cherng · Cloud Foundry Summit Silicon Valley 2017
• 45mModern Data ArchitectureAlexey Grishchenko · JavaDay Kyiv 2015
• 25mAccelerating Data Processing and Consumption for Building Modern Apps at the EdgeMarilyn Basanta, Ed Byford · Kubernetes on Edge Day North America 2022
• 53mBig Data: from mammoth to elephantRoman Nikitchenko · JavaDay Lviv 2015
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 41mApplying Geospatial Analytics at a Massive Scale using Kafka, Spark and Elasticsearch on DC/OSAdam Mollenkopf · MesosCon North America 2017
• 30mFive Challenges in Real-time Stream Processing and Five SolutionsFawaz Ghali · SREday 2023 London
• 01h 20mSupercharging Operations and Analytics: Using Spring XD to Support Analytics and CEPJoseph Paulchell · SpringOne2GX 2015