Just because I simplified the tasks graph by computing it in two different steps, which is OK for this simple case.
I did some more tests today, and I confirm this behavior. I tried to make the test smaller to debug, and saw that if I read just one Zarr file, then the problem disapears. I’m not sure if it comes from the concatenate step? I really don’t know what the problem could be, as the Dask graph looks good and trying to optimize it doesn’t change anything.
So I’m at a loss, and I would recommend to open an issue, this computation should be entirely streamable.