One output time vs multiple output time

Deploying Dask
1

Hi!

I have a function

@dask.delayed
def process_minute(m):
   ...

this function takes 1 minute to finish

to_compute = []
for m in pd.date_range(start=period, periods=1, freq='1min'):
       to_compute.append(process_minute(m))

computation, = dask.compute(to_compute, scheduler='processes')

However when using 8 tasks its taking 5 minutes

to_compute = []
for m in pd.date_range(start=period, periods=8, freq='1min'):
       to_compute.append(process_minute(m))

computation, = dask.compute(to_compute, scheduler='processes')

How can I interpret this result. It not suppose to do the 8 tasks at the same time and finished them in almost the same time that one task?

2

@juannaviap Welcome to Discourse and thanks for this question!

I can reproduce this. The performance drop you’re seeing is due to the “multiprocessing” scheduler. As mentioned in the docs here, the multiprocessing scheduler needs to transfer data between tasks which causes performance penalties.

I see you’re working with pandas, so the “threaded” or “distributed” scheduler will give you the expected performance (distributed scheduler is recommended though because it has nicer optimizations!):

import dask
import pandas as pd

from time import sleep
from dask.distributed import Client

client = Client()


@dask.delayed
def process_minute(m):
    sleep(5)
    return m


to_compute_1 = []

for m in pd.date_range(start="2022-01-01", periods=1, freq='1min'):
       to_compute_1.append(process_minute(m))

dask.compute(to_compute_1) # Distributed scheduler -- 5.16 s
dask.compute(to_compute_1, scheduler="threads") # Threaded scheduler -- 5 s


to_compute_8 = []

for m in pd.date_range(start="2022-01-01", periods=8, freq='1min'):
       to_compute_8.append(process_minute(m))

dask.compute(to_compute_8) # Distributed scheduler -- 5.03 s
dask.compute(to_compute_8, scheduler="threads") # Threaded scheduler -- 5.01 s

Let me know if this helps!