Hey, I’m trying to use Dask get()
to execute a DAG defined by a dictionary - The functions executed within the dictionary are also containing params
dictionary argument, which might contain string representations of another nodes of the graph (Even though it does not depend on them, and is used inside the function itself) when I’m creating the dictionary in such way, I see that I’m getting unwanted dependencies, Minimal reproducible example -
from typing import List
from dask.threaded import get
def inc(l: List, params: dict):
return l[0] + 1
def add(l: List, params: dict):
return l[0] + l[1]
dsk = {
"x": 1,
"y1": (inc, ["x"], {}),
"z1": (inc, ["y1"], {}),
"y2": (inc, ["x"], {"create_cycle!": "z2"}),
"z2": (inc, ["y2"], {}),
"final": (add, ["z1", "z2"], {})
}
get(dsk, "final")
y2
depends on z2
by the implicit dictionary param there. Is there a way to specify which arguments will be used for the dependency creation?