Using Dask as a DAG framework (no parallelization)

1

Hi,

At my job we have a computation cluster with a few 1000 nodes. We are allowed to submit jobs to the cluster using Univa Grid Engine (UGE), however each process is restricted to being single threaded and cap at 16 GB of memory. As such we must take great strides to limit the number of threads and memory numpy and the like can use.

Given these constraints we cannot use dask the way it is intended. It’s unfortunate but completely out of my hands.

There are however possible alternative use cases for dask such as being a DAG framework. Something we are investigating at work is switching from a more procedural programming style to a functional one. We currently have some production code with very large amounts of branching logic.

We would like to instead to swap to lookup tables of functions, selecting the functions from the look up table, based on some metadata, creating a task graph for Dask to interpret. Then use get function and let Dask figure out the topological sort, and compose the function calls together, and forwarding the outputs accordingly.

We initially tried to do this using dask.delayed but was incurring a great deal of overhead, as we were performing an anti pattern that the guides were explicitly warning against.

I would like to get the opinions on the efficacy of using the task graphs and get function in a single threaded environment with no parallelization allowed.

2

Hi @Delengowski, welcome to Dask Discourse forum!

Well, I’m not 100% certain about this assumption. With Dask and dask-jobqueue package, the idea is to launch one job per Worker, so you could have several Workers each taking one process and one thread and use a Dask cluster, couldn’t you?

Could you give us a link of where this anti pattern is documented?

Well, you can perfectly do this. It’s a little hard to answer without more details on your workflow. For example, as a parallelization framework, Dask introduce overhead for each task, about 1ms latency, so if each of your task are really short, this might not be a good solution.