Getting heartbeats from unknown workers

Hello, I’m getting a bunch of these errors running Dask:

2022-07-20 10:39:51,552 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:44235'. 
2022-07-20 10:39:51,552 - distributed.worker - ERROR - Scheduler was unaware of this worker 'tcp://127.0.0.1:44235'. Shutting down.

And then the process ends – it’s fine with one thread, and I’ve ensured there are no other python processes listening on the network at all. The Dask version I’m using is 2022.7.0.

My code is basically just:

if __name__ == "main":
    with Client() as client:
        fire_and_forget(client.map(really_long_process, list_of_inputs)))

Okay, I got a bit further with my EC2 cluster.

  extra_bootstrap = [
      "DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y awscli s3fs",
      "aws ecr get-login-password --region [region] | docker login --username AWS --password-stdin [registry]",
  ]
  with EC2Cluster(extra_bootstrap=extra_bootstrap, n_workers=mp_num, docker_image="[registry]", docker_args="--device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined",  # For mounting S3 buckets
                  instance_type="[instance]", security=False,  # To work around https://github.com/dask/dask-cloudprovider/issues/249
                  iam_instance_profile={"Name": "[iam_profile]"}, worker_options={"nthreads": 4}) as cluster:
      with Client(cluster) as client:
          client.register_worker_plugin(AddProcessPool)  # From https://www.youtube.com/watch?v=vF2VItVU5zg&t=515s
          with dask.annotate(executor="processes"):
              client.gather(client.map(my process, my_input))

It can’t seem to find the "processes" executor on the worker, which means that the register_worker_plugin call isn’t propagated in this case.

@hameer-spire Welcome to Discourse! Could you please share a minimal+reproducible example, and some more details on how you’re setting up your cluster in the first example? It’ll allow us to help you better.