Processing 2.7 million images with Elixir (vs Ruby)
summary
A project reprocesses 2.7 million images originally handled by CarrierWave and S3 and finds Ruby’s one‑second per image approach would take ~31 days, threads drop that to ~18 days, while an Elixir OTP app using Ecto, HTTPoison and magnify plus an Erlang S3 library and a poolboy + supervisor architecture reaches about 0.128 s per image and finishes in roughly four days of run time though the full effort takes ~16–18 days including integration work.
AI-written summary · about
chapters (9)
- 0:00Intro
- 9:45Creating the app
- 12:08Downloading images
- 14:59Uploading images to Amazon
- 20:44Processing per image
- 21:50Poolboy
- 26:45Conclusion
- 27:41Elixir is so great
- 31:22Outro
related talks
30mYour own 'Images as a Service'
06mThumbnail Generation in Elixir/Erlang with libVIPS
28mFaster websites: integrating next-gen images in your...
37mCache is King: Get the Most Bang for Your Buck From Ruby
32mProcess Data Streams Efficiently in Elixir
22mFrom 3000ms to 300ms: Lessons learned from optimizing Tanstack ...
33mI Made a Slow Thing Fast
44mWriting fast Ruby