I’m trying to understand work stealing, with the plan to allow running workers in different datacenters, but prevent any but essential transfers between them. I expected to be able to prevent stealing on a case-by-case basis by overriding _can_steal in stealing.py . I set up an experiment where a s…

[image] gjoseph92: you could also probably write a custom optimization function which traverses the High-Level Graph and propagates the resource annotation from parent tasks to children I wrote a little utility to do this. I haven’t tested it on real clusters at all, but I’d be curious if it wo…

More experiments reveal that _can_steal is only called if _has_restrictions has already returned True. Removing that requirement (it’s only an optimisation) I can now control work stealing for each task and worker, at least for a customised copy of stealing.py. I’ll look at putting together a PR

@dmcg Thanks for the question! I believe this is related to Multi-region clusters . Would you be able to share a minimal version of your experiment using LocalCluster? It’ll be super helpful for someone in the future! If I understand correctly, you’re looking for a way to allow work-stealing within…

Yes, this is part of some work on distributed data across datacenters, regions or organisations, and having the compute tied as far as possible to the place where the data resides. Instead of using workers we are (ab)using resources. So each worker in an organisation is assigned that organisation a…

Hey @dmcg this is such an interesting and unique use case! I’d love to set up a time to learn more about it and see how our oss dask engineers at Coiled might be able to give you some tips. Are you free to talk sometime this week or next? Here’s a link where you can find a time that works best for…

That would be grand - I’ve booked a slot

And in the meantime submitted a PR Remove duplication from stealing (was Allow subclasses to control work stealing) by dmcg · Pull Request #5787 · dask/distributed · GitHub

[image] dmcg: This works well (provided all the workers can see each other) but doesn’t prevent work stealing between organisations If you have resource restrictions on tasks, but those tasks are getting stolen to workers that don’t have appropriate restrictions, that sounds like a bug. Work …

Thanks all. On further investigation we don’t see resource restrictions being ignored for the initial data loading tasks (these are not stolen). We could annotate downstream tasks to ensure that they are not stolen, but our trials show that with our modest change to the stealing, they don’t have to…

So, I now remember why this scheme works well even though the ‘downstream’ tasks are not annotated. It’s because our org_aware_can_steal takes no notice of the task when deciding if a worker can steal, it only uses the resources attached to the worker to establish whether or not they are in the same…

Thanks for the detailed experiment @dmcg . It seems like your tweaks do make your particular test case work. I’m still not sure though that they’ll address other scenarios. [image] dmcg: We could annotate downstream tasks to ensure that they are not stolen, but our trials show that with our mode…

Understanding Work Stealing

Distributed

dmcg March 10, 2022, 9:42am 13

Thanks so much for taking the time to do this (and pool is such a better name than org ;-), I’ll try it out.

Topic		Replies	Views
Scheduler not saturating workers Distributed future , distributed	9	353	August 9, 2023
Multi-region clusters Distributed	3	516	February 8, 2022
How to retry hanging jobs during a distributed computation Distributed dask-array , distributed	3	993	May 4, 2022
Adaptive Scaling while not rerunning non-idempotent tasks Distributed	8	278	December 22, 2023
Optimising Dask computations (memory implications and communication overhead) Distributed delayed , future , distributed	6	352	October 12, 2023

Understanding Work Stealing

Related topics