How to append to a Dask Dataframe

[KonGian]

Thank you for your help.

@guillaumeeb

This build a Dataframe in a lazy way.
However, I tried to show an example from your code, but there is a big limitation here: the values to be put inside each partitioned DataFrame are computed in the main thread, which is not good if you’ve got a lot of data. If you want to really go distributed and avoid memory pressure on the main thread/process, you would need to compute the values inside delayed calls too.

My intention is my code to be fully distributed. I am new to Dask and distributed computing so as you mentioned I maybe have plenty of mistakes in my code.

@guillaumeeb

You’re saying you’ll need a Dask Dataframe because you’ll have too many combinations to store. But that means that this code:

@KonGian

combinations = list(itertools.combinations(ddf.columns, 2))

will probably won’t be able to store all the combinations into memory. And computing a list of unique combinations in a distributed way is hard

You are right about that. Could you please provide me a code that will be fully distributed and addresses the problems you mentioned? I am not sure how i can compute the values inside the delayed calls.

@guillaumeeb

Also, how will you build your input DataFrame? Can you read your input data by chunk using Dask?

I have a csv as input so I use dd.read_csv.

Thanks again.