Hi @hmacdope, welcome to this forum!
I have to admit I’m a bit surprised: I didn’t find any real documentation about dask-jobqueue and GPU.
First, you don’t need to use cuda workers to use GPUs on your system. Plenty of users already use Dask with GPU on HPC systems without it. It could just simplify the use of GPUs. You’ll need a Python environment with Cuda and Cupy though.
We discussed this a bit in Using dask distributed scheduler with CuPy.
I’m not sure if there is currently a way to use DaskCudaWorker with dask-jobqueue. Maybe through specifying Worker class when configuring your JobQueueCluster.
But I see you also contributed some code, thanks!! I’ll try to review it in the following weeks.