Growing data in a Ruby world
summary
Project handles over a billion contacts, moves from MySQL to a hybrid stack: Cassandra as source of truth for scalable writes, Elasticsearch for searches, Kafka (Confluent) for messaging, and Sidekiq for async indexing. Tried Alessandra and DataStax but rejected for maturity and cost. Prototype existed in Node then rewritten in Ruby. Final decision is synchronous writes to Cassandra and async Sidekiq updates to Elasticsearch. Deployment is still experimental and not yet proven in production.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:37Agenda
- 1:08What Is Gem
- 1:32Email Marketing
- 9:37Alessandra
- 12:10Kafka
- 13:43The Confluent Platform
- 18:39Elasticsearch
- 22:06Production
related talks
26mRuby APIs for NoSQL
45mRuby for the soul of Big Data nerds
29mScaling Rails to 2M MySQL requests per second
11mSpinning up micro-services using Ruby/Kafka
05mRuby startup demo – Nagendra Dhanakeerthi
30mRuby in the Billions
35mProcessing data at scale with Rails
33mThe rise and fall of the full stack database