Scheduler memory management

[guillaumeeb]

Hi @cbritogonzalez,

@cbritogonzalez

Is there a way to require the scheduler to use a certain amount of memory?

AFAIK, no, there isn’t. There is a discussion about it here: Placing limits on scheduler memory with some tips.

@cbritogonzalez

I do know I have an issue with the size of the dask graph that are sent to the scheduler (some graphs are up to 800mb in size and then deserialized it is even bigger)

And do you know why you have such a big graph?

@cbritogonzalez

Further more, I do remove any local references to dask futures and I also make use of the .cancel() function to help with this as well.

This should allow Scheduler Python process to free some memory, are you sure you don’t keep any reference to some data or Future? Without some code snippet or MVCE, it is hard to help more here.

The amount of memory Scheduler use is directly linked to task graphs or Futures kept in Cluster memory.