Web Development To Big Data: A Journey
summary
A web developer moves to a production big data stack and claims systems now handle £100,000 per hour by replacing brittle Python ETL on one Windows box with a data lake on S3, Parquet files, Airflow DAGs and Apache Spark for batch and structured streaming. Storage and compute are separated with Presto/serving clusters and BigQuery for fast ad hoc queries; a custom BigQuery connector for Spark and a simple collaborative recommender are built and deployed, reducing nine‑hour jobs to minutes and adding alerting and schema validation.
AI-written summary · about
chapters (26)
- 0:00Sam I Am
- 1:02Why I Changed Careers?
- 2:20WTF Is Big Data
- 4:18Redshift Datawarehouse Amazon Redshift works with your analysis tools
- 6:11Immutable Infrastructure
- 6:39Slow Updates
- 8:05Run Every 15 Mins
- 8:58New Sources = Pain
- 11:07Building A Data Pipelines NOBODY BUILDS
- 11:29Make Data Queryable! SHOW ME
- 13:06Architecture!
- 14:13Handle Bad Data
- 15:31Schema Updates!
- 16:01Alerting!
- 17:10Data Formats
- 17:42Columnar Storage
- 18:30Data Lake
- 20:07Apache Spark
- 22:13Data Serving Layer
- 23:16Lambda Architecture
- 23:35Data Streaming
- 23:52Issues With Streaming Wait a minute!
- 24:52Structured Streaming AKA V2 Continuous Application
- 25:48Google BigQuery
- 27:43ML Powering Recommendations Recommendations.
- 29:15DEMO!
related talks
35mData Processing Pipelines for Small Big Data
58mUnderstanding Big Data for Software Engineers
11mBreaking Data Silos: Enterprise Data Management
41mData engineering in 10 years
33mData: The Land DevOps Forgot
01h 35mThe Modern Data Stack
02h 05mBig Data and the Data Lake from the Analyst Perspective
30mBig Data and Small Data Everywhere, All at Once