Dask_mpi.initialize memory_limit setting

How to set the memory_limit, when use dask_mpi.initialize().

On an HPC, I have 4 nodes. Every node has 180GB and 36core.

When I set desk_mpi.initialize(nthreads=2), the output log shows that
distributed.worker - INFO - Memory: 10.41GiB

But it should be 90GiB, ideally. Here the default setting is “auto”.

I have read this intro, but still do not set memory_limit

Thank you

Hi @MichaelChaoLi-cpu, in the link you posted, you can see:

memory_limitint, float, or ‘auto’

Number of bytes before spilling data to disk. This can be an integer (nbytes), float (fraction of total memory), or ‘auto’.

So just try memory_limit='180GiB', or memory_limit=1.0.

In your case, memory_limit should be 180GB or close to that value, you’re launching only one worker process per node.