Migrating Apache Flink data pipelines to write to Azure Cosmos DB (Cassandra API)
summary
Apache Flink + Kafka pipelines move from on‑prem Cassandra to Azure Cosmos DB Cassandra API using AKS and Java; the Flink Cassandra Sink relies on Datastax v3 unless a custom v4 connector is built. Migration uncovers SSL setup, rate‑limit failures (jobs die), and throttling by Request Units; fixes include the open‑source Cosmos Retry Policy and a constant reconnection backoff ≈2000 ms. Partitioning limits 10,000 RU per physical partition, four replicas per region, CQLSH copy for small sets and Flink jobs used in place of Spark for bulk migration.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:11The Story Current
- 2:39Cosmos DB (Cassandra API)
- 5:04Know your DB Resource Model
- 6:47Get going with tools
- 7:35Untangle Connector APIs & SSL
- 11:08Dealing with Request Units (RUS) for Capacity Request Unit : Cost of De operations expressed as a performance currency abstracting the system resources such as
- 13:27Leveraging Insights Drill down analysis of all DB operations by time range to monitor
- 14:09Optimization & Error handling Handling rate limiting (429 errors)
- 18:42Planning for Availability Replica or node outages mitigated with 3 replica of data in each Azure region
- 20:02Plan Data Migration Choose based on throughput requirements & tech stack/skills.
- 20:32References
related talks
07mDeliver In-flight stream processing with Apache Flink on Azure
14mPower real-time data streams from anywhere to Cosmos DB with Confluent
43mContinuous SQL with Kafka and Flink
43mA Data Streaming Architecture with Apache Flink #bbuzz
37mConnecting Apache Flink to the World...
41mBeyond the Hype: When Apache Flink Solves Real Problems
45mApache Flink — Real-Time Data Processing
26mRunning Apache Flink on AWS