Cleaning Up Errored Futures

Distributed
1

Hi,

Does anyone here have advice on cleaning up errored futures in a Distributed dask cluster? We have a long running distributed cluster we basically use as a web backend. When tasks error, they are held for a long time in memory as failed futures (along with their dependencies). We would like to clean up failed tasks so that they’re dependencies aren’t held in memory if they are no longer needed by anything else. Any help is greatly appreciated!

2

Hi @joecus1,

Did you try what is explained in the documentation?

Dask will only compute and hold onto results for which there are active futures. In this way, your local variables define what is active in Dask.

You can also explicitly cancel a task using the Future.cancel or Client.cancel methods

Does it help in cleaning memory?

3

Hi, we are not holding the errored futures in memory, so don’t think that would cause us to retain them. We’ll try cancelling the futures after they error and see if that helps, thanks!