I have created a scheduler and cluster using dask python advance APIs. Now I want to scale the workers based on scheduler load but none of the APIs is there for that. The only way is from the cluster API.
Any idea how to join make scheduler join the cluster or any other way how we can scale workers while we are using dask python advance APIs.
@NamitPathak Welcome to Discourse!
In the Python API (advanced) doccs, I see that the Scheduler has functions like
workers_to_close, etc., that can be used to scale up/down workers. Would these help, or are you looking for something else?
If you can share a minimal example of your workflow, I’d be happy to test out scaling.