Linear-time blockwise layer fusion

See dask-awkward/optimize.py at main · dask-contrib/dask-awkward · GitHub for a function that does fusion on chains of blockwise layers in linear time, rather than the current N**2 algorithm in dask. It is more limited in what it can fuse, but could be an effective precursor stage to standard fuse as a way to cut down time.

Since it was developed for dask-awkward, it might only apply when there is just one axis of partitioning, i.e., dataframes rather than arrays, because this matches dask-awkward’s model.

1 Like