Hi, after a “relatively” resource-demanding dask.compute(list_of_delayed_objects) on a dask_gateway-cluster, (on this google-cloud-deployment), I want to start a much smaller task. I am looking for a way to only let a small number of the remote workers start that task, while the others wait. I am guessing this is possible, but have not found the method yet.
What would be a good way to, for example, tell 3 workers to do the second computation below, without scaling the cluster down?
cluster.scale(20)
parameters = dask.compute( list_of_timeconsuming_delayed_tasks )
save( parameters, ... )
cluster.scale(3)
results = dask.compute( list_of_less_timeconsuming_tasks)
Edit: I was previously looking at this method. It is not a problem for me to client.submit instead of computing delayed objects, so if that helps then I can easily change that. But I am not sure if this is the correct method to use, and I am unsuccessful in specifying the resources on the remote cluster when following the examples.
