Consider an SSHCluster: in the “worker_options” and “scheduler_options” we can specify the “local_directory”.
Consider this path: ~/Apps/Apps1
from what I can see in Terminal, this is the same as: /home/user1/Apps/Apps1
I would like to use the ~ in the paths (rather than /home/user1), because the “home” directory is different on different machines in my cluster, and SSHCluster does not allow setting different settings for different workers.
If I specify: /home/user1/Apps/Apps1 as local_directory, it works, i.e. it creates and uses the following as local directory: /home/user1/Apps/Apps1/dask-scratch-space
but if I specify: ~/Apps/Apps1 as local_directory, Dask seems to be creating a folder, named “~” in /home/user1. Within the ~ folder, Dask creates the dask-scratch-space folder. This is not a huge problem, but it is not what I intend (I expect the dask-scratch-space folder to be created in: /home/user1/Apps/Apps1)
If I omit the local_directory, it seems to work, but I don’t find the dask-scratch-space folder anywhere. What does it default to in such a case?
The SSHCluster constructor also accepts “remote_python” as one of its parameters. In this case: ~/Apps/Apps1/bin/python3.11 works like a charm. (there is no python3.11 in /home/user1/~, hence it wouldn’t work if it wasn’t pointing to the correct file).