Using Dask Distributed with numpy.f2py function in module

Distributed
1

Hi,
I’m trying to use dask distributed to distribute a custom module across multiple workers. The module uses numpy.f2py, and I’m getting errors running it:

client.upload_file(‘/…/EAP_02/EAP_tn.py’)

/nobackupp11/tjnorman/dask-worker-space/worker-jxynhyrj/EAP_tn.py in
11 sourcecode = sourcefile.read()
12 f2py.compile(sourcecode, modulename=‘Dmmex_R14B_4’)
—> 13 import Dmmex_R14B_4
14 import numpy as np
15 #import pandas as pd

ModuleNotFoundError: No module named ‘Dmmex_R14B_4’

Where Dmmex_R14B_4 is a fortran program being converted for use in python by numpy.f2py.
Does anyone has any suggestion on how to solve this problem?

Thanks

2

Hi @tjnorman and welcome to discourse! As you’ve noted in the error message, the Dmmex_R14B_4 module is not accessible to the workers. In addition to the file /…/EAP_02/EAP_tn.py’, have you also already uploaded Dmmex_R14B_4 to the workers? Additionally, if you want to upload a more comprehensive module you can package it into a zip or egg file - see Futures — Dask documentation for more details.

1 Like
3

Thanks @scharlotte for your response. I tried your suggest and am encountering similar errors and same problem. When I use client.upload_file(‘/…/EAP_02/Dmmex_R14B_4.f’) to upload the file or upload all the relevant modules as a zip file I get the error:

ModuleNotFoundError: No module named ‘Dmmex_R14B_4’

Dmmex_R14B_4 is a fortran file. Can Dask read fortran files?
Thanks

4

Hi @tjnorman thanks for the update!

Ah I see I think I misunderstood your question, I thought you had already converted Dmmex_R14B_4 to a Python module using numpy.f2py and were looking to load that module to the workers. Instead it sounds like you’re hoping to upload a fortran file to the workers, which will not be interpretable by Dask. When I try to do this, I get a warning: distributed.utils - WARNING - Found nothing to import from fib1.f:

from dask.distributed import Client, LocalCluster
cluster = LocalCluster()
client = Client(cluster)
# created from https://numpy.org/doc/stable/f2py/usage.html example
client.upload_file('fib1.f')

Apologies for the confusion!