How to monitor GPU usage using dashboard

I’m a relatively new user of dask. I’m using dask on a GPU cluster. On the CPU cluster I’ve been able to monitor CPU usage using the dask dashboard. I see no analog for GPU usage. I have been monitoring usage directly by logging into the nodes and using cuda/nvidia cmd line tools but if there is a graphical tool people are using or even a cmd line dask tool that routes to the correct nodes for monitoring, that would be very useful. Any advice?

Hi @liberabaci, welcome to Dask Discourse forum!

There is a great post about this by @jacobtomlinson, and certainly he also knows more than me about default monitoring of GPU included in Dask.

There is support for GPUs in the dashboard. It requires the nvidia-ml-py package to be installed. If you are using dask-cuda this will be installed automatically.

1 Like

@guillaumeeb @jacobtomlinson Thank you so much for the info. Exactly what I was looking for.

1 Like