How Python Drives Modern Data Engineering
summary
Shows concrete pipelines: Python scripts using PySpark on AWS Glue or Dataflow, loading into Redshift or BigQuery after ETL/ELT, and streaming user events via Pub/Sub to meet sub‑10s SLAs. Demonstrates hybrid migration patterns, data masking/encryption choices, and that Apache Beam/Spark plus SQL let local development before cloud rollout. Cites a Gartner projection of 51% public cloud spend shift by 2025 and gives sample architectures that scale to millions of events and actually deliver timely order confirmations.
AI-written summary · about
chapters (14)
- 0:00Python in data engineering
- 2:37Why Python is best
- 5:11Data roles explained
- 7:58Modern data engineering
- 13:16Python in on-prem systems
- 16:50Python cloud services
- 17:55Cloud vs on-prem
- 19:24Business use cases
- 23:39Time-sensitive pipelines
- 27:12Python career growth
- 27:55Beginner tools
- 30:47Security in data engineering
- 33:40Startup first steps
- 35:46Data roles breakdown
related talks
48mThe Use of Python in the Data World
27mModern ETL-ing with Python and Airflow (and Spark)
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
50mBig Data Processing and Data Science Environment in Python
44mUsing Python with Hadoop to create production ready big data applications
38mData processing with Apache Beam
01h 22mIntroduction to Big Data Processing using Spark and Python
34mPython as a Framework for Analytics and Growth Hacking