Distributing Work With Queues and GCP PubSub
summary
Claims a queue-based redesign using GCP Pub/Sub and Broadway splits each user search into per‑airline work items (often tens of megabytes of XML, sometimes ~40 MB XML), publishes one message per airline, and uses filtered subscriptions keyed by pod name plus a base64 ETF caller‑PID to route results back. Workers run as Broadway processors with tuned concurrency; moving CPU work to a worker pool flattens CPU spikes and reduces P99 latency to match prior sync paths. Memory GC issues appear; workaround spawns short‑lived tasks so heaps free on exit.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:41Flight Search Crash Course
- 6:48Searching Multiple Airlines
- 12:40Selecting a queue system
- 13:57GCP PubSub Basics
- 18:30Message format
- 19:39Enqueuing messages
- 20:35Routing back to process
- 23:22Architecture
- 23:49Broadway tweaking
- 25:40The first version was slow, tweaking made a huge impact
- 27:20Benefits of the new design
- 28:45Memory issues with Broadway
- 30:01Erlang VM Memory Management
- 31:42Bug in GCP PubSub Provider
related talks
40mDistributing Work With Queues and GCP PubSub
37mUsing Elixir for scheduling cloudnative DAG workflows
45mKEYNOTE: Announcing Broadway
34mRunning Large-Scale Stateful Workloads On Kubernetes at Lyft
56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...
19mStarburst: A Cost-aware Scheduler for Hybrid Cloud
36mBackground Jobs with Elixir: Oban in the Real World
14mMaximizing Aerospike Administration in GCP