Increasing data size leads to the error in dask-gateway

Distributed
, ,
1

I am using dask-gateway based on the Kubernetes cluster. I wrote a code which executes successfully for the some size of data. After increasing the size of data, I am getting the following error: ‘in <TLS (closed) ConnectionPool.gather local=tls://172.17.0.210:22133 remote=gateway://{getway_Ip}:80/default.f835e074e78746319404c55407528d75>: Stream is closed’.
I would be happy if someone could help me.

2

Hi @arthurlalayan,

I would hazard that your worker died, probably some OutOfMemory error considering it fails when you increase the size of the data. You should try to find the Dask-worker logs to see if this is the case, or maybe follow the computation using Dask Dashboard.

3

Is there any way to estimate how much RAM I need (minimum ammount less than that will certainly lead to the OutOfMemory error) to overcome the issue? I want to process large-scale data (lets say 1 TB) using dask distributed.

4

It really depends on your workflow. Without a Minimum Working Example, or at least some code showing what you’re trying to accomplish, it will be hard to tell.

You might also be interested in a new setting introduced in Dask recently: Share your experiences with `worker-saturation` config to reduce memory usage · Discussion #7128 · dask/distributed · GitHub.

5

thanks my issue has been fixed.

6

thanks, it works for my case.

1 Like