Dask worker using 600% of the CPU

Hello, I’m using Dask on a HPC (slurm). I just started running some compute-intensive models on dask workers. However, weirdly enough, I see a cpu usage of about 600% on the worker which is running the models (screenshot attached). I wanted to know how dask is calculating that number or what exactly (like number of threads) is getting it up so high. This is likely not a bug in the release because we have modified the source code to suit our needs. If anything, we probably messed up when modifying the SlurmCluster class but that’s why I just wanted to know how dask calculates to that number so that I can go ahead and fix it. Thanks!

Hi @sash19, welcome to Dask community!

I didn’t find the source code, but I think this comes from psutil, monitoring the worker process CPU usage on the system. Since your process has several threads, it can potentially use up to 800% CPUs, meaning 8 full cores (or virtual cores). So I don’t think this is a bug, it is just system metrics.

However, the Total cpu usage is wrong, it should either be the sum of all usages, either we should reduce it to a percent of thread level usage.

cc @crusaderky.

Code here:

Calculation of CPU% on the total line:

and I agree it makes little sense.

Thank you so much for looking into this! I think this will help.