Hello,
I am a newish dask user that was confused by some of the documentation for the scale method to jobqueue Clusters. The “How this works” page currently says that
For the full computation, you will then ask for a number of jobs using the
scalecommand:
I took this to mean that cluster.scale(n) defaulted to cluster.scale(jobs=n), but it looks like scale actually scales the number of workers you request. This resulted in me getting fewer jobs than I intended when I had multiple workers per job. Likewise, its not very clear from the SLURMCluster documentation that n refers to the number of workers, and not the number of jobs in the scale method.
I wasn’t sure if this was information was described elsewhere and I missed it or if this language was intentional, so I thought I would post here before opening up an issue on github. Let me know if this is something we can clarify.