Multiple local clusters on the same (virtual) machine: PermissionError

Distributed
1

Hi,
in our team we are using Dask LocalCluster while connected via ssh to a Google Cloud.
The problem is that if there is more than one script running at the same time, errors appear, like:
“PermissionError: [Errno 13] Permission denied: '/tmp/dask-worker-space”

  • Is there a way to launch separate localclusters on the same machine and ensure they work independently? I tried setting different cluster names but it doesn’t seem to do anything.

  • What would be a better setup to have multiple people using Dask on the same machine? (I am aware that it’s possible to deploy a Dask cluster on Google Cloud but I was looking for a quick and easy fix that doesn’t impact our workflow much.)

2

Bumping in case anyone has an answer

3

Known issue: Putting `dask-worker-space` in `/tmp` is unfriendly on shared systems · Issue #6748 · dask/distributed · GitHub
Feel free to contribute a PR

4

You can probably work-around this using:

cluster = LocalCluster(name="MyCluster", local_directory="/tmp-MyDirectory")

Mix that with some os.environ[‘$USER’], or any random generated hash to be sure to have a unique worker space directory.

5

Issue will be fixed in release 2022.10.0