How can I let only a few workers start a small task, while others wait? (on a remote cluster)

Hi, after a “relatively” resource-demanding dask.compute(list_of_delayed_objects) on a dask_gateway-cluster, (on this google-cloud-deployment), I want to start a much smaller task. I am looking for a way to only let a small number of the remote workers start that task, while the others wait. I am guessing this is possible, but have not found the method yet.

What would be a good way to, for example, tell 3 workers to do the second computation below, without scaling the cluster down?
parameters = dask.compute( list_of_timeconsuming_delayed_tasks )
save( parameters, ... )
results = dask.compute( list_of_less_timeconsuming_tasks)

Edit: I was previously looking at this method. It is not a problem for me to client.submit instead of computing delayed objects, so if that helps then I can easily change that. But I am not sure if this is the correct method to use, and I am unsuccessful in specifying the resources on the remote cluster when following the examples.

Hi @ofk123,

If using Future API is possible, then maybe this would do the trick:

You just need to get 3 workers address first.

1 Like

Thanks alot that works! :grinning: