From time to time, some of the tasks stack in a “processing” state. It looks like the task was never reported back as “completed.” Is there any way to configure “max execution time” and kill/restart the task in such cases?
Any ideas are welcome.
The same application with an older version of Dask has an issue with “Waiting tasks.” at some point, the Dask scheduler shows a constant number of “waiting” tasks, which never goes down unless the scheduler and workers are restarted.
Hi @mkazitov, welcome to Dask community!
Which Dask API are you using? What is your workflow? Would you be able to share a reproducer?
Do you by any chance launch tasks from tasks?
I use the latest Dask version.
Using
client.submit() + client.map()
And yes, I start tasks from tasks, and this part looks suspicious to me.
The problem is pretty hard to repro. Usually, it happens after many circles of execution. But on the prod scale, it is repro every day.
Launching task from task can lead to deadlock. Not saying this is your problem, but did you read through Launch Tasks from Tasks — Dask.distributed 2024.4.1 documentation? Particularly the secede and rejoin functions?
It looks like the task from the task is the root of the issue. After removing this feature, the issue disappears.