# Reduce topologies for general datastructures
**URL:** https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037
**Category:** Uncategorized
**Created:** [July 19, 2023, 10:14pm UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037 "2023-07-19T22:14:49Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@vij](https://dask.discourse.group/u/vij)
#### Post date: [July 19, 2023, 10:14pm UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037/1 "2023-07-19T22:14:49Z")
I see tree reduce mechanism implemented for dask arrays [1](https://github.com/dask/dask/blob/59da6fa0c425faa30827cef37958e13d2e823d98/dask/array/reductions.py#L256). I currently operate on dask Bags and I was curious if there are existing dask reduce implementations which allow for custom reduce operations ?
* * *
NOT A CONTRIBUTION
---
### Author:  [@vij](https://dask.discourse.group/u/vij)
#### Post date: [July 19, 2023, 10:20pm UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037/2 "2023-07-19T22:20:53Z")
I guess this is one implementation : [dask.bag.Bag.fold — Dask documentation](https://docs.dask.org/en/stable/generated/dask.bag.Bag.fold.html?highlight=reduce#dask-bag-bag-fold)
I was curious if there are existing benchmarks for different kinds of reduction topologies with dask datastructures.
* * *
NOT A CONTRIBUTION
---
### Author:  [@guillaumeeb](https://dask.discourse.group/u/guillaumeeb)
#### Post date: [July 20, 2023, 11:58am UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037/3 "2023-07-20T11:58:38Z")
So as you’ve seen, there are Map/Reduce like operations implemented in all the high level collections that Dask offers.
I don’t think there is any benchmark, but as Dask Bag is a collection made over Python object, this is generally considered as less optimized than Arrays or Dataframes which rely on Numpy and Pandas which are C optimized libraries under the hood.
See [Bag — Dask documentation](https://docs.dask.org/en/stable/bag.html#shuffle:)
> These shuffle operations are expensive and better handled by projects like `dask.dataframe`. It is best to use `dask.bag` to clean and process data, then transform it into an array or DataFrame before embarking on the more complex operations that require shuffle steps.
---
### Author:  [@vij](https://dask.discourse.group/u/vij)
#### Post date: [July 20, 2023, 2:33pm UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037/4 "2023-07-20T14:33:51Z")
In this case I am mostly manipulating GPU hosted `torch.Tensor` objects. They exist in this form as most of the transformations utilize torch’s cuda kernels which are not available in the cudf or other Rapids libraries. However for the reduce operation in question I can utilize `cudf` operations.
Would you know if the overhead of transforming `dask.Bag[torch.Tensors]` → `dask.Array[float]`, only for the reduce operation, is worth the speed gains? My experience with performing `torch.Tensor` mini-batch processing in `MapPartition` paying the conversion cost from dataframes → Tensor for every user-function makes me believe that the conversion overhead is high.
* * *
NOT A CONTRIBUTION
---
### Author:  [@guillaumeeb](https://dask.discourse.group/u/guillaumeeb)
#### Post date: [July 20, 2023, 2:50pm UTC](https://dask.discourse.group/t/reduce-topologies-for-general-datastructures/2037/5 "2023-07-20T14:50:35Z")
> [@vij](#):
>
> Would you know if the overhead of transforming `dask.Bag[torch.Tensors]` → `dask.Array[float]`, only for the reduce operation, is worth the speed gains?
Sorry, I can’t tell at all.