Is there any solution to trim workers memory on windows?

Distributed
1

Hi everyone,

I’m facing a problem concerning the memory not released back to OS. By inspecting the Dask dashboard (localhost:8787/status), I discover that each worker have a lot of unmanaged memory and I think that’s what causes the problem.

I have read this article and see that there is some solution to trim memory on Linux and macOS workers but nothing about Windows workers.

So, is there any solution to trim memory on Windows worker ? Or it’s just imposible due to the OS ?

2

Hello,
I’m the author of that article.
I’m not aware of anything like malloc_trim on Windows.

Are you sure you sure the problem you’re facing is not actually task heap memory? Do you still see the unmanaged memory after all tasks have stopped running?

Are you sure it’s not a leak? What libraries are you using in your tasks?

Does the situation improve with a larger chunk size?

Could you make a minimal reproducer for your problem?

3

@crusaderky Thanks for your response !

Indeed, with a larger chunk size the problem disappear.

I don’t think it’s a leak but I will check this anyway.