Data Acquisition: Overcoming Challenges
summary
Practical account of a working pipeline that streams production changes into a lake: Basium Kafka Connect to Apache Kafka, Apache Spark structured streaming, Delta Lake merges, Redshift Spectrum reads, AWS Glue and Lake Formation for schema and access, and DMS for snapshot restores. Automation via a Jira-triggered Airflow/Boto3 flow called Jarvis turns onboarding, consistency restores and masking into self-service. Achieves about 20 million events under two hours and peaks near 60 million under two hours.
AI-written summary · about
chapters (16)
- 0:00Data acquisition challenges
- 3:01Data platform goals
- 4:26Core challenges
- 5:50Implementing change data capture with Debezium
- 8:20Delta Lake merges
- 10:10Automated table addition
- 13:19Jira self-service
- 15:51Decentralized scaling
- 16:32Restoring consistency
- 18:22AWS DMS snapshot
- 20:25Airflow consistency
- 22:11Lake Formation privacy
- 24:18Automated privacy hiding
- 26:28Solutions summary
- 27:14Future plans
- 28:19Final thoughts & Q&A
related talks
37mBuild a Modern Data Platform on AWS
19mDemystifying Modern Data Pipeline Architecture
31mFrom Warehouse to Lakehouse: Data at Rest
47mBuilding Data Lake Platform Fully Serverless
31mHow to quickly build Data Pipelines for Data Scientists
30mData Engineering building your BI infrastructure from scratch
34mChasing the Stream Processing Utopia
31mDelta Lake: Open Source Reliability and Quality for Data Lakes