Understanding Work Stealing

[dmcg]

@gjoseph92

you could also probably write a custom optimization function which traverses the High-Level Graph and propagates the resource annotation from parent tasks to children

I wrote a little utility to do this. I haven’t tested it on real clusters at all, but I’d be curious if it works for you (would also welcome contributions):

GitHub - gjoseph92/dask-worker-pools: Assign tasks to pools of workers in dask

Thanks so much for taking the time to do this (and pool is such a better name than org ;-), I’ll try it out.