Memory limits reached in simple ETL-like data transformations

[guillaumeeb]

@theJonan

Btw, on your graphic I see a peak consumption of ≈1GB, is that correct?

You are right, not sure why. I just re run things with 300MiB memory limit, here is the graphic generated:

But on the Dashbaord, it did never go over 300MiB, so not sure why the memory scale is like that.

@theJonan

Setting process=True on my side actually makes the memory consumption some ≈150MB higher (understood)

Hm, I just looked at the Dashboard, did you ue some external tool?

@theJonan

this type of processing should comfortably go with around 100MB

Well I guess you’ll never get that with Dask.

@theJonan

Thanks a lot for your efforts! I’ll post here any updates, if I manage to make it work the way I’m looking for.

You’re welcome, please do so!