We wrote a framework that forces people to write logic as python functions. This made it really easy to offload, i.e. wrap these functions, to Dask for distributed computation. In particular, if you are using Pandas and use the API that Dask implements, you get seamless scaling without having to change any logic. See our hello world example - hamilton/examples/dask at main · stitchfix/hamilton · GitHub.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| How to handle a Dask DF in multiple modules? | 6 | 618 | February 8, 2023 | |
| Scaling Panel with Dask | 0 | 109 | January 19, 2024 | |
| Unable to create Dask dataframe at scale | 6 | 931 | October 22, 2022 | |
| Need help with efficient parallelization [local machine] | 2 | 288 | July 30, 2022 | |
| Dask.delayed and custom workflows | 2 | 341 | February 21, 2024 |