Terminate Workers when scheduler dies

Distributed
1

Hello,

Currently I have scheduler running on Machine A and few workers running on Machine B.

Assuming the workers are performing large computations which can take hours. When the I terminate the scheduler on Machine A, is it possible to automatically make the workers terminate immediately?

It seems like the current behavior is the workers will complete their task first then terminates.

Thanks,
Bill

2

Hi @bsun23, welcome to Dask Discourse forum!

How are you launching your workers? Using cli? Do you use the death-timeout option? Honnestly, I’m not sure if the Worker would be killed after this timeout if they are performing a task.

Another solution could be to cancel your task before terminating the Scheduler, can’t you do that?

3

Hi @guillaumeeb,

I am launching it via the python API.

I thought death-timeout options is only used for initial connection to the scheduler.

I could cancel the tasks, but there are quite a few of them so I was wondering if there is a quick and easy to kill all (by killing the scheduler)

Thanks

4

death-timeout is used also for connection lost. I’m just not sure of how much the Worker shutdown is graceful or not. You should try it, just add the death_timeout kwarg to your LocalCluster or Worker constructor.