At my job we have a computation cluster with a few 1000 nodes. We are allowed to submit jobs to the cluster using Univa Grid Engine (UGE), however each process is restricted to being single threaded and cap at 16 GB of memory. As such we must take great strides to limit the number of threads and memory numpy and the like can use.
Given these constraints we cannot use dask the way it is intended. It’s unfortunate but completely out of my hands.
There are however possible alternative use cases for dask such as being a DAG framework. Something we are investigating at work is switching from a more procedural programming style to a functional one. We currently have some production code with very large amounts of branching logic.
We would like to instead to swap to lookup tables of functions, selecting the functions from the look up table, based on some metadata, creating a task graph for Dask to interpret. Then use
get function and let Dask figure out the topological sort, and compose the function calls together, and forwarding the outputs accordingly.
We initially tried to do this using
dask.delayed but was incurring a great deal of overhead, as we were performing an anti pattern that the guides were explicitly warning against.
I would like to get the opinions on the efficacy of using the task graphs and
get function in a single threaded environment with no parallelization allowed.