Can a worker perform computation and io in parallel?

Distributed
1

Hi Dask developers,
I have a use-case where I want to run two tasks A and B in a single worker (both of these tasks are originating from different clients). Suppose A gets queued up at the worker before B and the result of task A is large in terms of memory (takes ~1s to transfer).

Is it possible to communicate the result of task A to the corresponding client in parallel while executing task B?. Basically I would like the io transfer to take place in one thread while computation can happen in a different thread in the worker.

Thanks in advance

2

Hi @proteetpaul, welcome to Dask Discourse forum!

I think that gathering result back to the client and computing tasks are orthogonal. client.gather can certainly happen while other tasks are being computed on the Worker. Is that your question?

We can probably verify this with a small example if needed.