JRuby and Big Data

on YouTube ↗41m

" summary

JRuby plugs into the Java Big Data stack: HDFS for fault‑tolerant storage, MapReduce for embarrassingly parallel jobs, HBase for low‑latency record access and ZooKeeper for coordination. Existing JRuby integrations (radoop, jruby-on-hadoop, papyrus) are stale; streaming (Wukong) and Cascading with JRuby support are viable. Concrete numbers: 155M tweets/day ≈ 16GB/hr (4MB/s) and a 10‑petabyte dataset was shipped on tape. File tech: Avro cuts 12MB JSON to 3.6MB and ZooKeeper needs at least 3 nodes for quorum; JRuby-as-native Mapper/Reducer support is work in progress.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 1:00Big Data Definitions
3. 2:25A lot of data
4. 5:15Lots of data
5. 6:50Big data
6. 7:10My personal term for big data
7. 8:04Why you need big data
8. 9:19When you dont need big data
9. 11:40hdfs
10. 12:08mapreduce
11. 14:52JRuby onhadoop
12. 20:54AVRO
13. 27:03Zookeeper
14. 29:15Bigtable
15. 30:13HBase
16. 31:23Data Logistics

" related talks

• 45mRuby for the soul of Big Data nerdsAbhishek Parolkar · RubyConf India 2012
• 31mExpand your Ruby Potential with JRubyCharles Oliver Nutter · RubyConf 2026
• 30mUsing JRuby: What, When, How, and WhyCharles Oliver Nutter · RubyConf 2022
• 49mJRuby in the WildKyle Maxwell · RubyConf 2007
• 39mJRuby on Rails: From Zero to ScaleThomas E Enebo, Charles Oliver Nutter · RailsConf 2019
• 55mJRuby in 2018: Building, Deploying, and Scaling Your AppsCharles Oliver Nutter · RubyC 2018
• 49mJRuby Everywhere! Server, Client, and EmbeddedThomas E Enebo · RubyConf 2016
• 34mBuilding Data Driven Products with RubyRyan Weald · RubyConf 2012