Distributing Work With Queues and GCP PubSub
summary
Searches hit airline APIs that can return 40 MB XML per request and vary wildly in CPU cost, so a synchronous API suffers high P99 latency. Solution uses GCP Pub/Sub to publish one message per airline and Broadway workers to pull and process them, with per-pod filtered subscriptions (pod name in metadata) and caller PID serialized in ETF/base64 to route results back. Result: matched pre-queue latency while greatly lowering P99 and evening CPU load; needed task-isolation for GC and revealed a Pub/Sub provider shutdown bug.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:41Flight Search Crash Course
- 6:48Searching Multiple Airlines
- 12:40Selecting a queue system
- 13:57GCP PubSub Basics
- 18:30Message format
- 19:39Enqueuing messages
- 20:35Routing back to process
- 23:22Architecture
- 23:49Broadway tweaking
- 25:40The first version was slow, tweaking made a huge impact
- 27:20Benefits of the new design
- 28:45Memory issues with Broadway
- 30:01Erlang VM Memory Management
- 31:42Bug in GCP PubSub Provider
related talks
40mDistributing Work With Queues and GCP PubSub
22mAsynchronous Computing at Scale: Overview and Learning
37mUsing Elixir for scheduling cloudnative DAG workflows
25mAvoiding Overload in Distributed Systems
59mBuilding resilient event-driven architectures
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
38mScaling Kube-Apiserver to Thousands of Tenants – Lessons Learned From kcp
12mThe Next API Frontier: Mastering API Consumption Across Distributed Envts