Run function in all worker

Distributed
#1

Hi, team,

In my case, I hope my function can run in each worker. Does pure=False make it work for this case?

The following is my code. The function “my_func” needs to be called in each worker. I have used pure=False in client.submit, and the variable my_worker_num is the number of my workers.

fs = []
for _ in range(my_worker_num):
        future = client.submit(my_func, some_argument, pure=False)
        fs.append(future)
results = applicant.gather(fs)   # Wait for all worker finishing

I have tested above code in my cluster which have 3 workers (my_worker_num=3). It looks every worker calls “my_func” once, but I don’t know whether this behavior is as designed.

Any comments is welcome.
Chris Ding

#2

@cuauty Could you please share some more context around what the function does and why you’d like to make sure it’s called on each worker?

If you’re looking into debugging things, you can check out client.run, which does exactly this.

But if you’re trying to track worker state with it, you can look into scheduler/worker plugins instead.

In general though, we recommend using the workers parameter in client.submit() to control which worker(s) do a computation.

Does pure=False make it work for this case?

Even though this might work for many cases, that’s not the intended purpose. pure=False just tells the scheduler that the function is impure, and it shouldn’t expect the function to produce the same result each time.