Dask worker using 600% of the CPU

sash19 · January 31, 2024, 10:30pm

Hello, I’m using Dask on a HPC (slurm). I just started running some compute-intensive models on dask workers. However, weirdly enough, I see a cpu usage of about 600% on the worker which is running the models (screenshot attached). I wanted to know how dask is calculating that number or what exactly (like number of threads) is getting it up so high. This is likely not a bug in the release because we have modified the source code to suit our needs. If anything, we probably messed up when modifying the SlurmCluster class but that’s why I just wanted to know how dask calculates to that number so that I can go ahead and fix it. Thanks!

guillaumeeb · February 2, 2024, 1:58pm

Hi @sash19, welcome to Dask community!

I didn’t find the source code, but I think this comes from psutil, monitoring the worker process CPU usage on the system. Since your process has several threads, it can potentially use up to 800% CPUs, meaning 8 full cores (or virtual cores). So I don’t think this is a bug, it is just system metrics.

However, the Total cpu usage is wrong, it should either be the sum of all usages, either we should reduce it to a percent of thread level usage.

cc @crusaderky.

crusaderky · February 3, 2024, 6:28pm

Code here:

github.com

dask/distributed/blob/4425516f86d7de9aed06053fc3e21a17fe20efc4/distributed/dashboard/components/scheduler.py#L4033


      
                  for ts in erred_tasks:
                      new_data["Task"].append(ts.key)
                      new_data["Exception"].append(ts.exception_text)
                      new_data["Traceback"].append(ts.traceback_text)
                      new_data["Worker(s)"].append(",\n".join(ts.erred_on))
                      new_data["Count"].append(len(ts.erred_on))
          
                  update(self.source, new_data)
          
          
          class WorkerTable(DashboardComponent):
              """Status of the current workers
          
              This is two plots, a text-based table for each host and a thin horizontal
              plot laying out hosts by their current memory use.
              """
          
              excluded_names = {
                  "executing",
                  "in_flight",
                  "in_memory",

Calculation of CPU% on the total line:

github.com

dask/distributed/blob/4425516f86d7de9aed06053fc3e21a17fe20efc4/distributed/dashboard/components/scheduler.py#L4303-L4308


      
          elif name == "cpu":
              total_data = (
                  sum(ws.metrics["cpu"] for ws in self.scheduler.workers.values())
                  / 100
                  / len(self.scheduler.workers.values())
              )

and I agree it makes little sense.

github.com/dask/distributed

Total CPU % on /workers tab makes little sense

opened 06:27PM - 03 Feb 24 UTC

crusaderky

good first issue diagnostics dashboard

From https://dask.discourse.group/t/dask-worker-using-600-of-the-cpu/2489/2 T…he CPU % on each individual worker scales from 0 to nthreads*100; e.g. on a worker with 8 threads it can go from 0% to 800%. This is coherent with several other CPU monitors in the wild so it makes sense. The CPU% on the Total row, however, is calculated as https://github.com/dask/distributed/blob/4425516f86d7de9aed06053fc3e21a17fe20efc4/distributed/dashboard/components/scheduler.py#L4303-L4308 So for example on a cluster with 2 workers, 8 threads per worker, if one worker is flat out busy while the other is idle, the page will show 400%, which makes very little sense. | name | nthreads | cpu | | --- | --- | --- | Total (2) | 16 | 400% | tcp://... | 8 | 800% | tcp://... | 8 | 0% | I think we should change the CPU% on each worker to go from 0 to 100% and that on the total line to do the same (total CPU usage across the cluster / total number of threads)

sash19 · February 8, 2024, 5:12pm

Thank you so much for looking into this! I think this will help.

Topic		Replies	Views
Very low CPU utilization on SLURMCluster Distributed distributed , performance	3	800	November 8, 2021
SLURMCluster on 64 nodes / Understanding Cluster scale method Distributed distributed	4	270	November 17, 2021
Dask distributed with very large queue Distributed	5	591	November 13, 2022
Dask distributed performance issues Distributed kubernetes , future , distributed	1	189	December 7, 2022
dask_jobqueue.SLURMCluster: multi-threaded workloads and the effect of setting "cores" Distributed dask-jobqueue , distributed	2	115	November 9, 2023

Dask worker using 600% of the CPU

Related Topics