Using a modern data stack to explore & visualize the impact of a global pandemic on airline traffic
summary
A functioning ELT pipeline lands OpenSky state vectors every 30 seconds to S3, yields about 2.5 billion points since late 2018, and after a 50% cleanup groups points into flight legs and flights; nightly dbt jobs in Snowflake + dbt + StreamSets run 90‑minute pipelines (incremental six days a week, weekly full refresh) and push star schemas to Power BI. Findings: outbound flights drop ~69% at trough, TSA throughput falls ~95%, cargo traffic holds steady, some carriers cease operations, and average cruise altitudes increase.
AI-written summary · about
chapters (11)
- 0:00Introduction to talk
- 1:23Pandemic impact on aviation
- 4:53Modern data stack vision
- 9:47Development pipeline overview
- 12:13Data source and architecture
- 15:41Extract load transform process
- 18:30About dbt and modeling
- 22:45Dimensional modeling approach
- 25:57Analyzing flight data trends
- 31:13Combining pandemic and mobility data
- 37:13About on-prem and Q&A
related talks
16mTidy Transit: Real Life Data Modeling for Public Transportation- 16mTidy Transit: Real Life Data Modeling for Public Transportation
30mIntroduction to Data Streaming
01h 28mA Guide to the Modern Data Stack
53mIntroduction to data streaming
22mData Quality with Great Expectations and Airflow in a Reverse-ETL World
26mData with a mission: a COVID-19 API case study
48mPractical experience of implementing Data Mesh in the air transport industry