MacOS: Distributed Client Dashboard stopped working

Distributed
1

Hi there
I run on macOS 12.3.1 with these versions:

Python 3.10.2 (main, Feb 2 2022, 05:51:25) [Clang 13.0.0 (clang-1300.0.29.3)]
Dask 2022.03.0
Dask Distributed 2022.3.0

I had the setup working perfectly. It stopped working without me knowingly updating anything.
I have two problems now

  • after starting the client and clicking on the link to the dashboard, the dashboard is not there. It seems as if the server does not come up on port 8787 at all
  • when trying to run without the client the system starts emitting warnings
2022-04-29 14:13:02,337 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 4.85 GiB -- Worker memory limit: 6.40 GiB

and finally kills the workers with

2022-04-29 14:13:58,808 - distributed.worker_memory - WARNING - Worker exceeded 95% memory budget. Restarting

Has anyone seen the same behaviour and can somebody advice to debug this further?
The output has not been to helpful so far.

2

@tomsen Welcome to Discourse, and thanks for this question! Could you please share a minimal, reproducible example of your workflow?

I’m guessing your computation needs more memory than available, and we’ll need to see your code to diagnose why.

In general, maybe something here can help: Managing Memory — Dask.distributed 2022.8.1 documentation

3

Hi @pavithraes thanks for replying. My first problem seems to be that the distributed Client does not come up properly and screws up the workers afterwards.

I really only start the Client first with

from dask.distributed import Client

client = Client(n_workers=4, threads_per_worker=2)
client

When opening the Dashboard URL displayed, I get an empty window, using curl I get a timeout. So I reckon, the Client does not come up properly (or the Dashboard server).

Afterwards when I do read my partitioned csv dataset, the workers behave badly.

BUT:
If I do not start the client at all, the rest of the code runs. So it seems the Client somehow messes up things. Is there a way to debug the Client to find out what is going on?

4

Update:
Today I upgraded to dask 2022.05.0 and things started to work again.
However I use IntelliJ IDEA 2022.1 (Ultimate Edition) and the builtin URL viewer does not seem to work.
If I open the dashboard URL in an external browser it works though and the memory problems have gone as well. :frowning: I have no clue what caused the problems and I am unsure how the version upgrade of dask fixed them.

1 Like