Run Dask Client in the Background

Distributed
1

I’m currently working on a Python Package / CLI which relies on dask distributed to run tasks on a given client.
I want to supply two different interfaces.
The first one would use the import dask.distributed and would e.g. by run in a Jupyter Notebook which keeps the kernel open.
For the second interface I’d like to have two CLI commands that would start and stop the Client.
Ideally you could then submit jobs via CLI to the client as long as it is running:

  1. mytool start <config>
  2. mytool submit <...>
  3. mytool submit <...>
  4. mytool stop

Is this possible / already implemented somewhere?
Currently I’m creating a client in every mytool submit cmd which prohibits me from submitting multiple jobs through multiple commands.

2

When you create a client, you can connect it to existing cluster so supposedly, you can create the cluster in (1), delete it in (4) and in between connect to the same cluster over and over.

You would need to keep track of the cluster that was created (or just keep using the same one). I am not sure what the outcome of submit should be - but you may need to use fire-and-forget tasks and somehow save the results to file or whatnot.

Note - I am just a simple dask user - so maybe there’s a better answer to what you are trying to do.

1 Like