Using ~ in scheduler/worker local_directory

Deploying Dask
1

Consider an SSHCluster: in the “worker_options” and “scheduler_options” we can specify the “local_directory”.

Consider this path: ~/Apps/Apps1
from what I can see in Terminal, this is the same as: /home/user1/Apps/Apps1

I would like to use the ~ in the paths (rather than /home/user1), because the “home” directory is different on different machines in my cluster, and SSHCluster does not allow setting different settings for different workers.

If I specify: /home/user1/Apps/Apps1 as local_directory, it works, i.e. it creates and uses the following as local directory: /home/user1/Apps/Apps1/dask-scratch-space

but if I specify: ~/Apps/Apps1 as local_directory, Dask seems to be creating a folder, named “~” in /home/user1. Within the ~ folder, Dask creates the dask-scratch-space folder. This is not a huge problem, but it is not what I intend (I expect the dask-scratch-space folder to be created in: /home/user1/Apps/Apps1)

If I omit the local_directory, it seems to work, but I don’t find the dask-scratch-space folder anywhere. What does it default to in such a case?

The SSHCluster constructor also accepts “remote_python” as one of its parameters. In this case: ~/Apps/Apps1/bin/python3.11 works like a charm. (there is no python3.11 in /home/user1/~, hence it wouldn’t work if it wasn’t pointing to the correct file).

2

Then, couldn’t you just use a relative path, e.g. Apps/Apps1? The folder should be created in the HOME directory.

Anyway, I don’t think it’s a good practice to use $HOME for a scratch space, /tmp would be better suited. I think the default is to create a temporary folder in /tmp (didn’t check though).