Seagull: A Distributed, Fault Tolerant, Concurrent Task Runner

on YouTube ↗34m

" summary

Seagull splits a 100,000-test monolith into timed bundles and finishes a full test suite in under ten minutes by running thousands of parallel executors on Apache Mesos and AWS Spot Fleet. It stores per-test p94 timings in Elasticsearch/DynamoDB, uses a greedy bin‑packing bundler with bundle-splitting on failures, and runs millions of ephemeral Docker containers daily while a custom autoscaler (CPU + runs-in-flight) keeps costs low — reportedly cutting infrastructure cost by ~80%.

AI-written summary · about

" chapters (28)

1. 0:00Seagull: A Distributed, Fault tolerant, Concurrent task runner
2. 0:56Outline What is Seagull? Why did we build it? Deep dive into Seagull Fleetmiser: Yelp's in-house cluster autoscaler Challenges and lessons learned Future of Seagull
3. 1:21Testing at Yelp Yelp needs to run-100,000 tests for its applications. Tests take-2 days to run if executed serially
4. 2:16Current seagull scale -350 seagull runs every day. Average runtime - 10-15 mins. -2.5 million ephemeral containers every day. Cluster scales from -70 instances to -450 instances. All spot instances -25 million tests executed every day.
5. 3:23Applications of seagull Test execution: unit, integration, acceptance, and selenium tests Locust: Yelp's load testing framework Photo classification: Classify tens of millions of photos in less than a day.
6. 7:32Seagull Mesos scheduler Written in pythong Uses libmesos One scheduler per test suite per run 50 schedulers running simultaneously at peak Customizable concurrency Fault tolerant
7. 8:56Placement strategies Aim: Optimize for seagull bundle setup time. Affinity for already used agents. Use as many resources in an offer as possible. This also simplifies the scale down.
8. 11:11Seagull executor Custom mesos executor written in python. Uses Mesos containerizer and group isolators. Does setup and teardown of bundles. Reports resource utilization stats. Uploads log files to 53, sends metrics to ElasticSearch and SignalFx.
9. 12:01Clusterwide resources Clusterwide resources: selenium and database connections Resources are not tied to specific agents. Zookeeper ephemeral znodes to keep track of how many connections are being used.
10. 14:16Log aggregation in splunk stdout & stderr of all the executors is stored in Splunk which allows us to see failure trends across multiple seagull runs.
11. 15:06Greedy Algorithm Test timings are stored in ElasticSearch P90 of test timings for last one week are stored in DynamoDB every day. The list is sorted in ascending order of test timings. Tests are packed into bins of 10 minutes.
12. 15:54Linear Programming algorithm Handle test dependencies. Some tests cannot be run together. Some tests need to run together We use the PULP LP solver.
13. 18:00Auto scaling strategies CPU utilization Seagull runs in flight
14. 18:15Based on CPU utilization Our tasks are CPU bound Autoscaler tracks the CPU utilization in the cluster, and makes decisions based on that. Scale up if the cluster utilization 654 for 15 minutes Scale down if the cluster utilization is 35% for 30 mins.
15. 18:36Based on the number of Seagull run submitted Autoscaler gets notified when a new seagullrunis submitted.
16. 19:18Scaling down is difficult! AWS Spotfleet does not allow us to specify which instances to terminate Autoscaler finds and terminates the idle instances, and readjusts the Spotfleet capacity
17. 20:46Bandwidth issues while talking to s3 Artifact and docker image download takes a long time causing seagull runs to be delayed.
18. 21:18Use VPC S3 endpoints Fast and secure access to 53 without any limitations on bandwidth Traffic does not leave Amazon network. *Caveat*: It can be only enabled for the S3 buckets in the same AWS region.
19. 22:16Central Docker registries get overwhelmed Setup: Multiple Docker registries on a single host behind an nginx proxy It failed to cope up with requests being made. Solution: Run Docker registries on every agent. Use /etc/hosts for address resolution
20. 23:47Spot instances AWS gives a warning 2 mins before reclaiming spot
21. 24:42Spot markets are volatile Fluctuations in spot prices of instances in certain markets can have an adverse effect on your application Getting the bid price right is hard. Trade-off between availability and cost savings.
22. 27:19Orphaned Docker containers Cannot kill containers because docker daemon gets busy which leads to orphaned docker containers. Containers take up resources that are not accounted for in Mesos.
23. 28:02docker-reaper Proxy for Docker daemon. Written in go. Forwards all the signals to its children. Cleans up all the containers after child process exits.
24. 29:27Mesos maintenance mode Designed to be used by a single operator Need external locking mechanism to make it work for multiple operators.
25. 30:23Scheduler improvements Use oversubscription Use task_processing library to replace the core-component of the scheduler. Use CSI plugin to implement clusterwide resources. Make it easier for other services/applications to use seagull for parallelizing tasks.
26. 31:30Executor improvements Containerize everything!!! Use Docker runtime in Mesos containerizer and eliminate the need to talk to Docker daemon
27. 32:33Autoscaler improvements More advanced autoscaling for better resource utilization Use multiple spot fleets. We may save more money? Use more instance types in the cluster.
28. 33:20We are hiring in Europe! • Offices in London or Hamburg, remote workers as welcome • Engineers or managers with dist-sys experience: Strong knowledge of systems and replication design Ability to work closely with information retrieval machine learning Strong understanding of operating systems, file systems and

" related talks

• 28mA Modern Application Builder’s Guide to TestingBikram Bora · AWS Summit Online EMEA 2020
• 28mA Modern Application Builder’s Guide to TestingBikram Bora · AWS Summit Online 2020
• 21mFrom Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional ContainersMatei Zaharia, Keith Winstein, Christos Kozyrakis, Francisco Romero, Qian Li, Sadjad Fouladi, Dan Iter, Shuvo Chatterjee · USENIX ATC '19
• 21mScaling Test Automation: Building Reusability for 10x Speed and EfficiencyMesut Durukal · CityJS Athens 2025
• 33mFrom Hours to Minutes: Run Thousands of CI Tests in Just MinutePramod Yadav · TestMu Conference 2025
• 29mEgad! How Do We Start Writing (Better) Tests?Andrew Knight · PyOhio 2018
• 35mElevate your testing: testing smarter, not harder, in the cloud!Laszlo Simity · Testing Talks Conference 2022 Melbourne
• 30mArchitecting High-Density Performance Testing with Container-Based SimulationShubham Bansal · DevConf.CZ 2026