When I a simple id_list = list(df['ID'].unique()) on a dask df, the tasks keep restarting. My df has 6000+ partitions. After 30 min of watching them complete the tasks, the dashboard goes white then restarts. No memory issue at all. It just restarts. Any idea why the tasks restart after finishing? Snapshot below:
@Hasna94 Thanks for the question, would you be able to share a reproducible example with some toy data? It’ll allow us to better diagnose what’s going on because a variety of issues can lead to this behavior.
Hi, thanks for answering the question. I solved it by increasing the number of workers. 
1 Like