Changing Worker Resources At Runtime

Hi,

I’m looking to see if I can change the assigned worker resources at runtime. So I can dynamically scale the number of workers that can handle handle certain tasks as workers go on/offline.

My thoughts were to currently do this by using a Plugin, either on the scheduler or workers. There does seem to be a Worker.set_resources function, though in that context, I don’t have access to the total number of other workers.

Making a SchedulerPlugin seems like a better place to do this. However, it doesn’t seem as though you can change the workers resources safely from here:

# scheduler_setup.py

class MyPlugin(SchedulerPlugin):
    def add_worker(self, scheduler=None, worker=None, **kwargs):
        state = scheduler.workers[worker]
        state.resources["new_resource"] = 1000
        print("Total workers:", len(scheduler.workers))
# main.py

def main():
    client = Client(asynchronous=False, scheduler_kwargs={"preload": "scheduler_setup"})
    sum_ = da.arange(10, chunks=2).sum()
    result = client.compute(sum_, resources={'new_resource': 1}).result()
    print(result)
    client.close()


if __name__ == '__main__':
    main()

This method will just hang, as the changes to resources from the scheduler don’t get populated out. Maybe there’s another function for doing this that I’m unaware of, or a way to get scheduler info from the WorkerPlugin instead.

Any thoughts on doing something like this?

@freebie Thanks for your question!

’m looking to see if I can change the assigned worker resources at runtime.

I don’t think so, resources are set when workers are created.

Based on your example though, looks like you do want to set resources when you create new workers?

Would you be allocating the same resource to the workers you plan on scaling or different resources to different workers?

If it’s the same, you can set an environment variable or use a global config, as mentioned in the docs you linked. That way, when you create new workers, they will automatically have that resource.

If it’s different for different workers, it might get tricky because I believe resources weren’t intended to be used dynamically. We can look into some workarounds though. Let me know!

Hi @pavithraes, yeah the second. I was looking at trying to scale them dynamically at run time. Reallocating resources around based on however many workers are available.

I guess this use case is not quite like saying ‘these workers have these resource e.g. gpu’, but more like ‘these workers can fulfil this role’. This is related to this other topic I posted before, and you had kindly responded to: High Availability and Resource Tracking - #2 by pavithraes

Best,
James