Dashboard latency

Distributed
1

Hello all,

Quick question that I hope someone here won’t mind answering… I am consistently running into latency issues with the scheduler dashboard that I am almost certain stem from a CPU bottleneck on the scheduler. However, the scheduler has access to 4 cores and only one seems to be utilized at ~100%. By latency I mean that the dashboard only updates every 5-10 seconds and navigating to another page takes 10-20 seconds to load.

My question is really two:

Is there anyway to enable the scheduler to run certain things asynchronously, therefore taking advantage of the extra CPU cores available to it and potentially reducing the latency of the updates?

Is this expected behavior and, if so, are there any best practices for keeping the scheduler in a state where it can update the dashboard relatively constantly?

Thank you so much

2

FYI - Solved this by reducing the number of tasks on the scheduler at any given time from 2 million to 20k.

Would still very much be interested in understanding if possible to get the scheduler to use multiple CPU cores. Or, if this is on the roadmap/something I could maybe try to help with.

3

Hi @secrettoad , welcome to Dask community!

As far as I know, the Dask Scheduler is single threaded, so it cannot take advantage of extra cores. I tried to look for this information into the documentation but was unable to find it though.

Scheduler bottleneck really is the number of tasks. In my experience, it is crucial to try to keep this number below one million and even 100 000 for the time being.

I think this has been discussed, but this work has never been done.

4

Hello,
The Dask scheduler is strictly single threaded. This is by design, to avoid the race conditions that are intrinsic in multithreading Historically, we had to deal with enough race conditions already that we really don’t want to add multithreading in the mix.

We are aware of performance degradation with 1mil+ tasks on the cluster. It’s in the TODO list.

In the meantime, I strongly recommend you don’t generate as many tasks. You can achieve this by (1) increasing chunk/partition size and (2) setting optimization.fuse settings to be more aggressive; e.g.

optimization:
  fuse:
    active: true
    ave-width: 16