Exploring Big Data with rubygems.org Download Data

on YouTube ↗24m

" summary

Uses BigQuery and the gcloud Ruby client to load and query rubygems.org download data, streaming or batch via CSV. Shows actual sizes — ~125k gems, ~900k download rows, ~750k versions, ~3.5M dependencies — and runs multi-gigabyte queries (demo 3.9 GB) in seconds. Reports concrete findings: top downloaded packages like rake and multi_json, rails not in the top five by raw counts, minitest outruns rspec, Rails 3 has more downloads than Rails 4, and many recent gems claim Ruby >=0 which looks suspicious.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and context
2. 1:25Overcoming big data fear
3. 3:41Exploring RubyGems data
4. 7:01Defining research questions
5. 9:11Introduction to BigQuery
6. 13:00Streaming data import
7. 16:24Batch processing data
8. 18:44Data analysis results

" related talks

• 44mAnthropology of Ruby: Excavating its GemsJeremy Hinegardner · RubyConf 2010
• 31mConfessions of a Data Junkie: Fetching, Parsing, and VisualizationBobby Wilson · MountainWest RubyConf 2011
• 31mExploring open data with BigQueryJen Tong · Devoxx Belgium 2017
• 29mBig Data Enterprisy Analytics App and RubyAmit Kumar · RubyConf 2011
• 26mBiggish Data With Rails and PostgresqlStarr Horne · Cascadia Ruby 2014
• 36mData Management with RubySergey Sergyenko · Øredev 2022
• 31mBiggish Data With Rails and PostgresqlStarr Horne · RailsConf 2014
• 30mBusiness Intelligence with RubyXavier Shay · MountainWest RubyConf 2012