Dask-jobqueue and SLURMCluster options

Deploying Dask
1

This is my first foray into the Dask world and I’ve hit a stumbling block when setting up my Cluster options. If I don’t specify the memory required I get the following error:

ValueError: You must specify how much cores and memory per job you want to use, for example:
cluster = SLURMCluster(cores=128, memory='24GB')

but if I do specify the memory required I get a complaint from the scheduler when I spin up workers:

sbatch: error: Job rejected: Please do not specify memory for node-exclusive jobs.

I’ve had a poke around - but can’t seem to find an obvious solution. Is there a workaround?

Many thanks

2

I’ve now stumbled across the solution here:

2 Likes