Rethinking the Existing Data Ingestion and Processing Pipeline
summary
A practical example of using pandas to implement an ETL pipeline: extracting data from multiple sources, applying validation, filtering and conversions according to business rules and internal representation, and performing intermediate joins with other sources as part of evolving a data-intake component.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:06Company overview
- 6:06Data processing issues
- 13:17New ETL requirements
- 20:10SQL to Pandas
- 27:21Pandas optimization
- 34:08Solution outlook
- 41:23Q&A
related talks
36mPython and ETL Schemas: The essential pivot for Artificial Intelligence
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
34mData Ingestion in Python using Apache Airflow
32mBuilding Your First Data Pipelines
44mIntroduction to Pandas
35mData Processing Pipelines for Small Big Data
35mData Warehousing with Python
01h 52mIntroduction to data analytics with pandas