What can you learn from thousands of source files in Github?
summary
Uses GitHub’s BigQuery dataset (3 TB, ~2.8M repos, 145M commits, 163M latest-file revisions) and SQL queries with regex-extract and quantiles to analyze Groovy/Gradle source. Finds under a million .groovy files totaling ~16M lines, 50% of files ≤37 lines and 90th percentile <148 lines, surprising frequent script names like A.groovy/B.groovy, common transforms led by @CompileStatic. Compares build systems (≈1M Maven vs ~500k Gradle files) and shows H2, MySQL, PostgreSQL as top DB drivers.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 7:43Google BigQuery
- 10:25How many Groovy files are there
- 12:04Most frequent Groovy file names
- 13:35Groovy scripts
- 15:05Saved queries
- 17:05Histogram
- 18:10Transformations
- 20:14Other interesting functions
- 22:17Maven vs Gradle
- 23:45Dependencies
related talks
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
14mGitHub's Data Story: Open Source at the Home for Open Source
32mCodebase Research
35mHow to Analyze Terabytes of Data from GitHub Archive at High Speed
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
31mExploring open data with BigQuery
24mUncovering Community and Project Insights Through Data Driven Methods
07mLightning Talk: Smaug's Cave: Studying Thousands of Gems at Shopify