I’ve recently started using Dask to parallelize some functions, but I have some doubts on how to use it properly and efficiently.
I’ll describe you the function I’m trying to parallelize, I don’t think it’s important the actual function, I just need to understand what to do.
I’m basically computing a metric on multiple images with their corresponding ground truths. Now, let’s consider a single image, which has multiple channels (so dimension is H x W x C). The metric takes as input the image and its ground truth, and then start computing “things” on sub-matrices of that image (let’s say, for example, 32x32 sub-arrays), and for each sub-matrices other “things” are computed on each channel. So, the code is something like:
def compute_metric(im1, im2): for i in range(channels): im1[:, :, i] = another_function(im1[:, :, i]) im2[:, :, i] = another_function(im1[:, :, i]) # other computations on the images ... ... return computed_metric def metric(image, gt, block_size, shift) for j in range(step_x): # number of sub-arrays on X dimension for i in range(step_y): # number of sub-arrays on Y dimension values[i, j, :] = compute_metric(image[j * shift : j * shift + block_size, i * shift: i *shift + block_size, :], gt[j *shift : j * shift + block_size, i * shift : i * shift + block_size, :]) return metric
So, basically, the function I call to compute the metric calls other functions to actually compute the metric.
Now, the images are really big and with lots of channels, so I was trying to parallelize the code. I tried to add the decorator delayed to the function
compute_metric and change the function
metric appropriately (so I’m not loggin the values in a matrix anymore, I’m just appending them to a list, and finally computing the sum), and finally launch it, but the computation was slower than the base code with no parallelization. How should I use delayd in this context? I mean, should I add the decorator to each function that is called by
Thanks in advance for the help!