Distributed Locust on AWS ECS not handling worker status updates

[nicklvsa]

Hello all 👋,

I've got a Locust setup that runs on ECS (a service for the master host and a service for the workers, each running on its own ECS task). Something I've noticed is that if one of the worker tasks gets killed or is replaced (by a deployment for example) the master service does not see this change. Instead, the worker count is incremented and the the master seems to think the previous workers are still reachable.

I didn't want to post this as a bug as it could be due to how we have the services configured, but wanted to see if other folks have seen this behavior and if there's anything I need to do in the workers themselves to ensure they shutdown properly (listening to the correct signals and reporting that signal being fired for example). Something else I've noticed while running this configuration is an issue with stopping the tests sometimes. I haven't quite pinned when this happens (seems to happen when more workers + tasks are running), but the issue prevents the test from stopping and the test gets stuck in a Loading state. I've seen similar issues in the repo that were closed / fixed, but it does seem to still occur with this setup. Curious if anyone may have also have any insight into why this might be occurring at scale.

For some added context, we're using ECS's service discovery for communication between the two services.

Thanks!
Nick.

[afaderoTextNow]

Hello, I seem to be having a similar issue as well. In our case we're also setup on AWS to run distributed tests we set the --processes environment variable to auto detect how many workers we need which comes to about 32. When running some tests with just 500 concurrent users with a ramp up of 10/s, we get an exception due to the master not being able to connect to a worker. This causes locust to hang and the only way to get round it is to restart the pods. Is there anything we should be doing to help us avoid this

2025-06-05 21:03:45,730] locust-master-695bbfd894-l25jm/CRITICAL/locust.runners: Unhandled exception in greenlet: <Greenlet at 0x7f4e5aa4e840: <bound method MasterRunner.client_listener of <locust.runners.MasterRunner object at 0x7f4e5ab7aea0>>> Traceback (most recent call last): File "src/gevent/greenlet.py", line 900, in gevent._gevent_cgreenlet.Greenlet.run File "/home/locust/.venv/lib/python3.12/site-packages/locust/runners.py", line 1119, in client_listener self.clients[msg.node_id].state = STATE_RUNNING ^^^^^^^^^^^^^^^^^ File "/home/locust/.venv/lib/python3.12/site-packages/locust/runners.py", line 625, in __getitem__ return self._worker_nodes[k] ^^^^^^^^^^^^^^^^^ KeyError: 'locust-master-695bbfd894-l25jm_52f86cf7067443eb9eb76fafbdd42120'

cc: @cyberw