[Dask on Ray] Dask do not respect Ray Cluster resources

1

Hello everyone,

TLDR: I have a custom resource assigned to Ray Cluster head/worker. I am using Dask with Ray scheduler and I want dask-with-ray to respect custom resources assigned to each head/worker inside Ray Cluster.

Of course, I am aware that there is something like: dask.annotate to specify custom resources for the job, but this seems not to work with Ray Cluster resources.

There is a Ray docs section Advanced Usage — Ray v1.9.2 on resource options, but I can’t make it work either with Dask and Ray.

Can anyone drop some hint where can I find some information on how to put some resource constraints/requirements on specific Dask operations that are scheduled by Ray?

2

@billwill Hi and welcome to Discourse!

You can think of dask.annotate as a suggestion Dask makes, that needs to be implemented by the Schedulers. In your case, I’m not sure how the dask-on-ray scheduler handles these.

Also, note that both Dask and Ray have the concept of “resources”, but I wouldn’t expect them to be interoperable because they may havee very different implementations.

I noticed that there’s a similar question on the Ray Community forum, and I think they might be in a better position to help you here.

1 Like