Graceful task termination

Distributed
,
1

Hello,
I have a running tasks that i submit to the cluster as a future:

def my_work():
    operationA
    operationB
    return C

try:
     f = client.submit(my_work)
     wait(f, timeout=10, return_when="ALL_COMPLETED")
except TimeoutError:
     f.cancel()

Suppose i submited my task and it is now at operationB and this operation takes too long and i want to cancel it. of course from client side i can just run f.cancel() as shown above but that would just kill the task and wouldn’t close it gracefully.
I am looking to implement something with signals that perhaps causes the future to raise an exception which intern causing operationB to stop and do other things to gracefully end the task completely.

is there a native way to achieve graceful task termination in dask?

2

Hi @jadeidev,

I looked for a solution, but unfortunately I cannot see one. I don’t think there is a way to do that natively with Dask.

3

Not at the moment.
There was an attempt about implementing this functionality in the past (Allow workers to cancel running task? · Issue #4694 · dask/distributed · GitHub), but was not terminated.

The best suggestion I can give you is to break down your task into smaller tasks, if possible. Dask will cancel the whole chain as soon as one task completes.