Workshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 2]
summary
Builds a working CLI extractor using Typer with Pydantic and Python type hints that accepts options for url path encoding and query and surfaces readable errors; handles CSV and parquet (once found a saved parquet with .csv extension), detects ISO‑8859 vs UTF‑8 encodings, and fails fast on bad args. Implements transforms: normalize column names, parse timestamps into hour/year/week, one‑hot encoding and season mapping, merges London and Washington DC tables (~17,000 rows) for downstream loading.
AI-written summary · about
chapters (6)
- 0:00Command line tools intro
- 5:40Command line tools with Typer
- 10:16Testing CLI with CSV files
- 22:53Data cleaning and schema
- 33:11One-hot encoding and location
- 41:29Merging datasets for warehouse
related talks
35mBad data - anecdotes and examples from the real world
26mText to Data: Make Your Code Malleable, Not Brittle
32mSimple Data Engineering in python 3.5+ with Bonobo
35mVisualizing research data: Challenges of combining different datasources
38mLightning Talks
40mArchitectures for Exploring Your Personal Data
03h 07mData of Unusual Size: Interactive Visualization
01h 34mKeep the Data Flowing