Ensuring Each Dask Task Starts on a New SLURM Job with a Limit of 5 Concurrent Jobs

Hi @dierkes-j, welcome to dask community,

There is currently no way to have dask-jobqueue start a new job for each task. This is clearly not the design goal of Dask and hence dask-jobqueue.

A need close to this one has been discussed in Restart cluster job on task completion · Issue #597 · dask/dask-jobqueue · GitHub.

There are also other tools like GitHub - facebookincubator/submitit: Python 3.8+ toolbox for submitting jobs to Slurm who might be better suited for this.