On my dashboard, I am seeing the memory usage on the cluster and each individual worker fluctuating (< 20%), but no task is being processed. What could be some possible explanations of this?
Could this be caused by a
.compute() or by
client.scatter() on a large array? Currently I have
filtered_waves = filtered.compute() filtered_da = da.from_array(filtered_waves,chunks=wave_on_slice_channel.chunks) filtered_futures = client.scatter(filtered_da, broadcast=True)
I am fairly certain that the top
.compute() is completed. I suspect very much that the code is stuck somewhere in between the second and third line, i.e. by the
However, before this section of code, I have done exactly the same with
wave_future = client.scatter(wave_on_slice_channel,broadcast=True)
wave_on_slice_channel as well as
filtered_da have exactly the same shape and size (~ 11 GB).
My individual workers each have 100 GB and the cluster has > 2 TB of memory.