ECSCluster adaptivity

Distributed
4

Hey!, thanks for the replies, I actually got the cluster working by using the ECSCluster from the dask-cloudprovider like @jacobtomlinson mentioned. I did manage to maintain my deployment on CDK and just reference the scheduler address. However, I did have a bit of trouble since some of the documentation was conflicting IMO (or I just misunderstood). This is what I ended up needing to specify to get things up and running (using dask version: 2025.01.0):

ECSCluster(
                region_name="ecs-cluster-region",
                scheduler_address="scheduler-address",
                scheduler_task_definition_arn="scheduler_task_definition_arn",
                worker_task_definition_arn="worker_task_definition_arn",
                fargate_workers=True,
                fargate_use_private_ip=True,
                worker_nthreads=1, 
                worker_mem=1,
                cluster_arn="cluster-arn",
                execution_role_arn=execution_role_arn,
                task_role_arn=task_role_arn,
                cloudwatch_logs_group="your-log-group",
                security_groups=security_group_ids,
                vpc=vpc_id,
                subnets=subnet_ids,
                skip_cleanup=True,
                worker_extra_args=[
                    "--worker-port",
                    "9000",
                    "--nanny-port",
                    "9001",
                    "--nworkers",
                    "1",
                    "--no-dashboard",
                ],
            )

Now the tasks just spawn into the cluster and not into the service but that is fine, the implementation works well. Thanks for the comments and feedback @guillaumeeb and @jacobtomlinson!

1 Like