Can't see dask dashboard using docker on remote machine with reverse proxy

Distributed
,
1

I’m running dask on a digital ocean machine with docker containers.

The problem is that I cannot access the dask dashboard.

The setup:


> Docker-compose:
> 
>   scheduler:
>     image: ghcr.io/dask/dask:latest
>     platform: linux/amd64
>     hostname: scheduler
>     ports:
>       - "8786:8786"
>       - "8788:8787"
>     environment:
>       - USE_MAMBA=true
>       - EXTRA_CONDA_PACKAGES=xarray s3fs boto3 scipy matplotlib python-dotenv
>     command: ["dask-scheduler"]
> 
>   worker:
>     image: ghcr.io/dask/dask:latest
>     platform: linux/amd64
>     command: ["dask-worker", "tcp://scheduler:8786"]
>     environment:
>       - USE_MAMBA=true
>       - EXTRA_CONDA_PACKAGES=xarray s3fs boto3 scipy matplotlib python-dotenv
>     deploy:
>       replicas: 4
> 
>   notebook:
>     image: ghcr.io/dask/dask-notebook:latest
>     platform: linux/amd64
>     ports:
>       - "8888:8888"
>     environment:
>       - USE_MAMBA=true
>       - EXTRA_CONDA_PACKAGES=xarray s3fs boto3 scipy matplotlib python-dotenv
>       - DASK_SCHEDULER_ADDRESS="tcp://scheduler:8786"
>       - JUPYTER_TOKEN=$PASSWORD
>       - CHOWN_EXTRA=/home/jovyan/seamod
>       - CHOWN_EXTRA_OPTS=-R
>     user: root
>     volumes:
>       - .:/home/jovyan/seamod
>     working_dir: /home/jovyan/seamod
>     restart: always
>     depends_on:
>       - caddy

Caddyfile has a reverse proxy set up:

dask.seamod.xxx {
  reverse_proxy scheduler:8786
...
}

jupyter.seamod.xxx {
  reverse_proxy jupyter:8888
}

I can access the jupyter server, but I cannot access the dashboard. The dashboard shows up blank, and I try to initialize it via:

cluster = LocalCluster()
c = Client(cluster)
c

Client

Client-1735331d-a99c-11ee-8840-0242ac190007

|**Connection method:** Cluster object|**Cluster type:** distributed.LocalCluster|
| --- | --- |
|**Dashboard:** http://127.0.0.1:8787/status|

Any help would be greatly appreciated so I can move onto to troubleshooting how to manage memory within my workers…

Thank you!!!

2

Hi @turbid_butterfly, welcome to Dask Discourse!

Here you are setting a proxy for the Scheduler, but you didn’t print any line for the Dask Dahsboard on port 8788, is this normal?

I’m not sure what you are trying to do here, this would create a new Dask LocalCluster in the Jupyter notebook container, but what you want is to connect to the one you spawned in Docker containers, isn’t it?

3

Here you are setting a proxy for the Scheduler, but you didn’t print any line for the Dask Dahsboard on port 8788, is this normal? - I’m not sure?

…

Right, I want to connect to the one I’ve already spawned in the Docker container. I’ve also tried:
Client(‘tcp://scheduler:8786’)

4

I’m not familiar with Digital Ocean setup, but if you want to access the Dashboard running inside a Docker container, you should probably add some proxy for it too. Considering the docker-compose setup, it should be available on the 8788 port.

And what happens then?

5

http://127.0.0.1:8787/status

If I’m running in Jupyter, it says “Unable to connect”:

unabletoconnect_jupyter
unabletoconnect_jupyter1588×837 65 KB

I thought this reverse proxy was for the dashboard:

dask.seamod.xxx {
  reverse_proxy scheduler:8786
...
}

since it connects to the scheduler container…

6

So I assume the notebook your into is the one deployed from your docker-compose setup.

With the same docker-compose, you also started a Dask cluster with a Scheduler and 4 Workers.

In your notebook, you create another Dask cluster inside the notebook container (LocalCluster line), and you connect the client to it (since your passing the cluster object to the Client, I’m not sure the URL after is used at all). This cluster is not accessible from your local computer, you would need to use jupyter-server-proxy to see it, but I don’t think this is what you want.

It is for the connection between your Client and the Scheduler, the dashboard is on port 8787 by default. And you make it available on 8788 in your docker-compose setup.

I think you want to connect the Client to the already started cluster, so only something like:

client = Client('tcp://scheduler:8786')

Then maybe the Dask Jupyter extension can connect to scheduler:8788, but this might be a bit more complicated looking at your setup. On what URL do you open the Jupyterlab?

7

The dashboard is on port 8787 on the jupyter container?

I open jupyter lab on jupyter.seamod.xxx.com, where xx is the domain name of my company that’s hosting the web service.

If I run:

  client = Client('tcp://scheduler:8787')   
  client

It times out:

OSError: Timed out trying to connect to tcp://scheduler:8787 after 30 s
8

Dashboard is on port 8787 on the scheduler container, but considering your docker compose, you make it available to 8788 from outside:

The Dashboard is a web application, it has to be opened in a browser. Client is meant to connect on port 8786 on the Scheduler.