Using Streaming Data Pipelines in the Cloud to Achieve Success
summary
Optum Care builds a cloud streaming pipeline that ingests databases via Kafka Connect (Debezium/JDBC), uses Avro + Confluent Schema Registry and producer compression, and increases JDBC batch_max_rows to 16000 to speed large initial loads. Lightweight Kafka Streams services standardize envelopes and stamp data quality rules (metrics to Grafana) rather than rejecting records. Delta/Databricks handles schema evolution and automatic flattening before landing into Snowflake/warehouse. Connectors remain single‑threaded for huge tables, so one‑time bulk loads are often done outside Connect.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:45Target Audience
- 2:31OptumCare
- 3:39Modernizing how the business usos data
- 4:49Tools & Architectures from the last 10 years
- 6:13Streaming Architectures
- 8:40First Principles
- 9:06Simplified Architecture & some options
- 12:18Kalka Connect example
- 15:10Streaming Platform
- 15:47Data Processing
- 20:01Standardized Metadata & Lineage Solution
- 23:28Streaming Data Quality Solution
- 25:00Example Data Pipeline
- 26:15Making sense of incoming COC streams
- 28:39Snowflake Streams
- 30:00Databricks Delta Lake
- 32:58Warehouse & Persistence
- 34:34Review
related talks
31mOne Data Pipeline to Rule Them All
31mBridge to the Cloud
39mData Streaming? I don't even know her
16mLife is But a (Data) Stream: Building Quality Data Pipelines
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
43mBuilding Scalable and Extendable Data Pipeline
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...