Stop FargateCluster from retiring workers

Distributed
1

I have a code that is similar to this:

dfs = []

for task in tasks:
    subtasks = gen_subtasks(task)
    df = (
        bag
        .from_sequence(subtasks)
        .to_dataframe(...)
        .groupby(...)
        .compute())

pd.concat(dfs).to_parquet(...)

My problem is that whenever the loop finishes an iteration, FargateCluster seems to retire all the workers, and then start them all over again on the next iteration.

Is there a way to keep the cluster alive until the end of the program?

As an alternative, I also tried to generate the subtasks as a dask task, but I couldn’t figure out how to do this. How could I integrate the whole process in dask?

2

I’m not very familiar with dask-cloudprovider, but @jacobtomlinson or @guillaumeeb might have thoughts on this. :slight_smile:

3

I’m not familiar with it either, but I don’t see a feature that would cause this without any reason.

@ian.liu88, could you give us the code you use to build and configure your FargateCluster? Do you use adaptivity?

4

Yeah it’s hard to help here without seeing an example of how you set up your FargateCluster.