Processing data at scale with Rails

on YouTube ↗35m

" summary

Builds a real Rails pipeline that ingests ~1M lobbying records by scraping zipped XMLs, unzipping to S3, fanning out jobs so each Sidekiq worker handles a single filing, and loading parsed fields into Postgres with ActiveRecord. Uses Nokogiri, HTTParty, rubyzip, the AWS SDK and Redis for last-modified caching, normalizes company names to avoid duplicates, and relies on a Postgres unique index plus Sidekiq horizontal scaling so most records load successfully while a handful fail for manual inspection.

AI-written summary · about

" chapters (14)

1. 0:00Intro
2. 0:38Data pipeline use cases
3. 2:25Open data is empowering to developers
4. 3:28Data pipelines and ETL's
5. 6:49Chipotle and DLA Piper in Lobby Focus
6. 13:34Download and unzip
7. 17:33Fanning out
8. 22:16Write filing to database
9. 24:16Idempotent pipelines
10. 25:50Normalizing data
11. 27:26Stuff changes
12. 28:55Scaling
13. 30:48Data pipeline options
14. 32:12Data pipeline opportunities

" related talks

• 31mBiggish Data With Rails and PostgresqlStarr Horne · RailsConf 2014
• 27mGrowing data in a Ruby worldTamar Rucham · Rubyfuza 2018
• 27mPayment workflow with Ruby and ClickhouseHarman Sohanpal · RubyConf India 2023
• 37mDenormalizing RailsDaniel Lucraft · Ruby Manor 2: Manor Harder
• 34mIndustrial RailsChristian Weyer · RubyConf Portugal 2015
• 31mMaking Data DanceClark Barrett · RailsConf 2015
• 26mBiggish Data With Rails and PostgresqlStarr Horne · Cascadia Ruby 2014
• 31mUsing Databases to pull your applications weightHarisankar P S · RailsConf 2018