@guillaumeeb thank you for your help. I have tried the suggestions you gave.
- Deleting the futures doesnt seem to do the trick here (i do wonder how does deleting the future from the client side affect workers?)
- I have also tried to reduce the cluster memory so i can experiemnt with memory usage, basically what happened is that the memory keeps filling and doesnt get reclaimed. it eventually saturates the cluster memory and workers cant proceed. see image below
- do you happen to know what does the nanny error mean "“Run out-of-band function ‘trim_memory’”?
