How to Analyze Terabytes of Data from GitHub Archive at High Speed
summary
GitHub Archive data at https://gharchive.org contains about 1.3 billion events and roughly 2 TB of JSON‑lines per year and wide events can exceed 400 fields. The recommended pipeline downloads hourly gz files and ingests them as-is into VictoriaLogs/VictoriaMetrics for fast analytics; sample queries run in tens to hundreds of milliseconds on a single Kubernetes node. A public VictoriaLogs demo playground holds 12 months of data and shows gaps and missing fields after an October outage.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:45GitHub Archive
- 6:00GitHub event types
- 12:16Analyzing GitHub data
- 17:34Relational databases limits
- 20:02Document databases issues
- 21:26ClickHouse issues
- 23:11VictoriaLogs for GitHub
- 26:05Demo and metrics
- 31:02GitHub Archive issues
- 32:46Conclusion
related talks
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
14mGitHub's Data Story: Open Source at the Home for Open Source
30mWhat can you learn from thousands of source files in Github?
29mExtract, Transform and Learn About Your Developers
41mStep By Step Guide to Building an Application Analytics System
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
24mPROCESSING 40 TB OF CODE FROM ~10 MILLION PROJECTS WITH A DEDICATED SERVER AND GO FOR $100