Web Development To Big Data. A Journey
summary
Migration from brittle Windows Python ETL to a distributed pipeline that actually works: move batch files into an S3 data lake, convert to Parquet with Spark, enforce schemas in code, and run Airflow DAGs to avoid 8–9 hour full-load jobs by using SQL Server change tracking + triggers for deltas. Replace self-hosted Kafka with Kinesis streams and Spark structured streaming with checkpointing; serve analysts via Presto/Athena or BigQuery (proof‑of‑concept cuts a $20k/month Redshift bill to a few hundred and answers 35TB queries in seconds).
AI-written summary · about
chapters (31)
- 0:00Introduction
- 0:43Sams background
- 1:47Why build APIs
- 3:04What is Big Data
- 3:21The 3 Bs
- 4:25ETL
- 5:03Data Warehouse vs Database
- 5:44monoliths
- 6:38immutable infrastructure
- 7:02slow updates
- 8:20invisible failures
- 9:20new sources
- 11:03data pipeline
- 11:17end goal
- 11:37architecture
- 13:46handling bad data
- 14:46handling schema updates
- 15:42alerting
- 16:30data formats
- 16:59columnar storage
- 17:44data lake
- 20:16DAG
- 21:43Presto
- 23:05Lambda Architecture
- 23:41Kafka
- 24:15Kinesis
- 24:50Spark Continuous Applications
- 25:39BigQuery
- 28:04Spark Connector
- 28:32ML
- 29:43Demo
related talks
34mTales of Data Architecture Evolution
30mData Engineering building your BI infrastructure from scratch
31mOne Data Pipeline to Rule Them All
31mHow to quickly build Data Pipelines for Data Scientists
01h 35mDeep Dive into the Big Data Landscape
41mOn Data: Sky above, sand below, peace within
31mA Journey of Migrating Data Platform from Data Center to Cloud
21mMigrate Data, Mesh in mind