Dask_mpi.initialize memory_limit setting

Distributed
1

How to set the memory_limit, when use dask_mpi.initialize().

On an HPC, I have 4 nodes. Every node has 180GB and 36core.

When I set desk_mpi.initialize(nthreads=2), the output log shows that
distributed.worker - INFO - Memory: 10.41GiB

But it should be 90GiB, ideally. Here the default setting is “auto”.
https://mpi.dask.org/en/stable/generated/dask_mpi.core.initialize.html#dask_mpi.core.initialize

I have read this intro, but still do not set memory_limit

Thank you

2

Hi @MichaelChaoLi-cpu, in the link you posted, you can see:

memory_limitint, float, or ‘auto’

Number of bytes before spilling data to disk. This can be an integer (nbytes), float (fraction of total memory), or ‘auto’.

So just try memory_limit='180GiB', or memory_limit=1.0.

In your case, memory_limit should be 180GB or close to that value, you’re launching only one worker process per node.