I’m also encountering the same issue. I have a similar setup:
- Create a LocalCluster with N+M workers
- Main thread creates N outer workers (futures) with (say)
priority = -10 - Each of these workers create M inner workers (futures) with
priority = +10
The idea is: Inner workers should finish first and return the results to outer workers, so they can work on them. After all inner worker’s tasks finish, the outer task can also finish.
But this causes deadlocks at the end, probably because the priorities got reversed. I’m not sure if it is per design, to make + priorities as high in user API, but use Linux-style internally, and this is what we see in the Dashboard (like provided by OP). If it is not by design, -10 becomes +10 and +10 becomes -10 somewhere - so deadlock can occur.
Deadlock case:
- 2 outer workers (say “O1”, “O2” tasks), 12 inner workers (say .“O1-123” style tasks, but many)
- The scheduler distribute them, so some workers have both outer and inner tasks
- At the end two workers remain unfinished (50% chance), each having two tasks:
Wx: O1, O2-234
Wy: O2, O1-543
So, if it were working correct, O1-543 and O2-234 with high priorities would finish first, then O1 and O2 can finish.
But because the priorities are not correct (?) somehow, O1 & O2 has precedence and the workers never finish because of deadlock.
In the next runs I’ll try them reversed…