Local Cluster with Two Nodes (Desktops)

Distributed
1

Hello all,

I’m a beginner Dask user here and was hoping for some advice.

I’ve successfully implemented Dask into some of the data analysis pipelines and got some pretty significant reductions in computational time on just a single machine. I had a few additional computers that I had wanted to add to my current computer to form a little cluster, but I haven’t been successful connecting any additional workers to the scheduler.

My configuration is set up as follows:

Primary Computer use as both scheduler and worker
Additional Computer as an additional worker

The two computers are connected on the same private network and both ping each other with no issue.

I’ve set up the same python version and environment on both computers

I set up the scheduler using a terminal on the primary computer:

dask-scheduler

Then on a different instance of a terminal, I also set the primary computer as a worker

dask-worker tcp://xxx.xxx.xx.xx:xxxx

The primary computer has no problems here and I am able to run my python scripts. The problem is when I try to add the new computer as dask-worker, I keep getting the error:

RuntimeError: Nanny failed to start

Anyone have any advice on how to trouble shoot this issue?

Thank you!

2

Taking a blind shot here, it looks like your python environment is horribly broken.
I suggest that you wipe it and regenerate it from scratch on both systems in a reproducible way, e.g. with a conda environment file or a pip requirements.txt.

Also, before the RuntimeError: Nanny failed to start there was a stack trace. If you want any kind of assistance from anybody, you need to post that.