Is there any way to find out what is the maximum number of clients a scheduler can connect to? I need multiple async clients with my use case and that is causes connection refused errors randomly, id there any relation with scheduler memory or cores with number of clients
what is the use case you need different schedulers for?
you can do it by creating them and giving them specific ids and just specify for each worker to which scheduler to connect to.
Worker API
you need scheduler_ip and scheduler_port
OK, so here the question is more a single Scheduler, and multiple Clients.
If you ask this, I guess you are creating a lot of Clients… Do you have any other inputs? How many, long living or ephemeral clients?
Scheduler is unfortunately monothreaded. Scheduler memory is needed for tasks, and also Workers, not sure for Clients.
Theoretically, there is no real limit, and I don’t think there is a way to know… But a Scheduler is definitly not designed for handling millions of clients…