No module named x found

I’ve been using notebooks with dask clusters and I wanted to start cleaning up my code and create modules (not so I can create libraries, but so I can pick apart my code into cleaner sections so I can rebuild it up from the modules and create a small version meant for export).

I can import my modules locally, but when I try to run a dask job that relies on a function from an imported module, it breaks.

I put my modules in a folder called imports

notebook.ipynb

\imports
functions.py
imports.py
constants.py

I’ve read a few posts and tried a few things (such as pythonpath and running everything from the same location). Tried creating a symbolic link from the root folder to where I have imports in a subfolder.

What is the best practice to load custom modules on a dask cluster? Is it importing a library the proper way (i.e. into the libraries folder of the venv)?

It seems like it should be a simple fix. If I hand code the function it works, but if I put it in a file for import, it doesn’t work in dask (but will work locally). I know it’s a namespace issue, but even if I put the same folder imports where it should be on the remote clients, still throws an error about a missing module.

@thistleknot Welcome to Discourse!

Could you please share more details about your cluster configuration?

If you’re running a LocalCluster, your workers should be able to access the modules directly. In case they are not, the complete error traceback will allow us to help you better.

And, if it’s a distributed cluster, then this is to be expected. You can use upload_file() to upload small modules to all workers. :smile:

Let me know if this helps!

1 Like