Hello there,
I am trying to use ucx for my multinode cluster. And I am wondering if dask-cuda is necessary for that. Can you provide please the steps to use ucx protocol in a dask cluster without necessary gpus.
Thank you
Best
@GueroudjiAmal Welcome to Discourse!
In general, you can use the “protocol” config option to use ucx, in the following ways:
dask-scheduler --protocol ucx, see: Command Line — Dask documentation (or)- set the “protocol” configuration using yaml, env variables, or
dask.config.set, see: Configuration — Dask documentation (or) - Add the
protocol="ucx://"keyword to your cluster class, as suggested here.
As per quasiben’s comment here:
You are correct you will need to add the
--protocol ucxbut you will also need a handful of env variables to enable specific devices such as inifiniband and/or nvlink. We started to outline those configurations here:
I don’t think so. Since this is a part of distributed and you’re not using GPUs, you wouldn’t need dask-cuda.
That said, note that the same discussion and this other issue also suggests ucx benchmarks for CPU only workflows aren’t clear yet. I’d encourage you to chime in on those issues directly if you have further question about what’s mentioned there!