Extract, Transform and Learn About Your Developers
summary
Built a pipeline that extracts GitHub data for four repos with Airbyte, loads into ClickHouse, transforms with dbt and visualizes in Metabase. Daily syncs hit GitHub rate limits, deduplicate logic creates a true stargazers table, and a memory bump on the ClickHouse cluster fixes large JSON normalizations. Dashboards show issues, PRs, week‑over‑week trends, a high stargazer→issue→PR funnel, and one user touching all four projects; Metabase cards are also synced back into ClickHouse for query auditing.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:00Extract and Load
- 2:45Data Warehouse
- 3:26Analyze
- 4:44Airbyte
- 10:19Metabase
- 11:06Demo
- 23:03Increase the memory
related talks
24mUncovering Community and Project Insights Through Data Driven Methods
17mBuilding Real-Time Data Lake with ClickHouse
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
34mUncovering Community and Project Insights Through Data Driven Methods
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
28mExperiments in Backing Prometheus with Clickhouse
18mIntroducing a SQL-based metrics layer powered by