Activity Stream Processing in Clojure

on YouTube ↗41m

" summary

Prototype using MongoDB with embedded interested-user lists fails under large fanouts, causing timeouts. Switching to Redis sets and sorted sets still chokes on runtime unions, so a Clojure daemon keeps feed heads in memory using Clojure atoms and threads and writes to Redis. Replacing build-time work with a Storm topology (spouts, bolts, Storm spouts bolts DRPC) moves union/score work off the hot path, cuts Redis storage by about 95%, and yields faster, more reliable feed builds at the cost of RAM and orchestration complexity.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 27:59raw processing power receive tuples from streams
3. 38:04easy parallelism
4. 39:09mature, idiomatic library support

" related talks

• 46mClojure Scaling the Event StreamDerek Troy West · Clojure/conj 2017
• 42m"Distributed stream processing in Ruby"Walerian Sobczak · RubyC 2019
• 31mClojure at 4,000 msg/s-What We Learned, Loved, and LoathedNathan Barnett · Clojure/west 2017
• 26mMetrics Stream Processing Using RiemannPradeep Chhetri · OSMC 2022
• 35mApplying the paradigms of core.async in ClojureScriptJulian Gamble · Clojure/conj 2014
• 48mTaming & Composing High Performance Stream-Oriented Processing SystemsRiccardo · Functional Conf 2017
• 37mDon't Shed a Tier for Me, ClojureDavid Pollak · Clojure/conj 2013
• 30mRealtime Collaboration with ClojureLeonardo Borges · EuroClojure 2015