I am interested in using Dask inside a python package. In an ideal case, the package should use available resources (local cluster, HPC, others) without too much intervention from the users.
I am reading Dask tutorials, examples, and forums. However, I could not find a good discussion on this topic so far. Is there any good pattern to include Dask in a python package? Should we have a Singleton Pattern to initiate the scheduler and use that instance anytime we want to connect that client to that object? Is it better to inherit it, or pass it as a global variable? Thanks.
@Naeemkh Hi and welcome! This could indeed be a good discussion. I’m wondering if some more details about the package you’re working on might help scope this discussion?
Also, have you seen: https://stories.dask.org/ – maybe something here is similar to your use case?
2 Likes
Thanks, @pavithraes. As far as I could manage to review these examples, I could not find a good pattern of using Dask inside a Python package. One good example is the xarray package that uses Dask for internal parallel computing. I was wondering if it is the standard practice of using Dask inside a python package.
1 Like
This issue is related to this discussion:
2 Likes