Dask Plugin for after finishing execution of a dask graph

Distributed
1

Hello there and anticipated thanks for the help!

We are currently trying to leverage dask gateway in our compute provisioning solution. The dask graphs executed are more like definitions of jobs, for which we would need the results stored for audit purposes.

Currently we could leverage that the client requesting for the results to also be the one that stores them in out persistence, but i have recently learned of the Dask Plugin capability and was wondering if it could fit the bill.

The RabbitMQ example makes me understand that we could use plugins to connect to other services, f.ex an Azure Blob Storage.

I also understand that i can also get the full task of the graph (the final node of the graph that is the key that we want to get results for), but i would like some help understanding how could i determine if the task is done/finished processing and how i could get those results before they are sent back to the client side as response.

Linking the plugin page documentation: Plugins — Dask.distributed 2024.1.0 documentation
Thanks!

2

Hi @RoxanaSpiridon, welcome to Dask community!

Do you think you would be able to write some code snippet of what you are doing now, and what you would really want to do?

I’m not sure if Plugin is the best solution for your need. Couldn’t you just add one more task at the end of the graph to do what you want on Dask cluster side?

I’m not sure if you could easily get the actual result of a Task from the Plugin APIs.