Some questions about SLURMCluster

5

Hi

Very sorry again for the late reply.

  1. For the memory part, yes it is quit a lot. Strange thing, when I set memory size to something around 10 GB, it starts complaining about memory spilling and worker exceeding %95 of memory budget or something like that. I don’t know why. I didn’t receive these problems when running the distributed scheduler on a local machine.
  2. The LocalCluster is implemented as a distributed dask scheduler running on a single machine with number of processes, and number of threads per process.
  3. For the performance_report when running dask.distributed integrated with SLURMCluster, I activate my Python environment first before running the main code. But as a precaution, I also activate the same python environment in the SLURMCluster configuration to be sure the workers are using the same Python environment.

Regards