Ruby in the Billions
summary
ClickFunnels moves a reindexing job that used to hit ElasticSearch for billions of rows into a Kafka pipeline written in Ruby so batch reads from Aurora MySQL publish JSON to Kafka and small Ruby consumers built with racecar and ruby-kafka push to multiple ES clusters in parallel while ActiveSupport instrumentation and New Relic track consumer lag Redis handles Redis throttling between stages and the process shrinks from 7 days to 20 hours saving IO cost and enabling replayable, real time sync.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker intro
- 2:29Ruby for processing billions of records
- 4:36Problem with database synchronization
- 8:51Introducing Kafka for scaling
- 15:03Ruby gems for Kafka integration
- 16:51Elasticsearch cluster basics
- 17:34Designing the search kick parallel process
- 20:50Recap of the solution and benefits
- 21:14Challenges and testing in Ruby
- 24:06Instrumentation and monitoring
- 27:22Lessons learned and results
- 28:03Why Ruby remains the best choice
related talks
31mEvent Streaming on Rails
45mRuby for the soul of Big Data nerds
42mSimplifying Logs, Events and Streams: Kafka + Rails
35mSo You’ve Got Yourself a Kafka: Event-Powered Rails Services
33mI Can’t Believe It’s Not A Queue: Using Kafka with Rails
27mGrowing data in a Ruby world
27mEvent Streaming Patterns for Ruby Services
42m"Distributed stream processing in Ruby"