Understanding Work Stealing

@dmcg Thanks for the question! I believe this is related to Multi-region clusters.

Would you be able to share a minimal version of your experiment using LocalCluster? It’ll be super helpful for someone in the future!

If I understand correctly, you’re looking for a way to allow work-stealing within each datacenter, but not between datacentres, right?

Overriding the private API (like with _can_steal) isn’t generally recommended… But, assuming you know which workers are in which datacenter, maybe you can set the workers parameters in Client.compute for the specific computations. Or, use annotations:

with dask.annotate(workers=[...]):

Would that work?