Access / re-initialize futures from multiple clients

Distributed
1

Hello,

Background

I recently upgraded from distributed==2023.7.0 to distributed==2025.5.1. Upon upgrading I found an interesting change that broke some of my workflows and looking to understand if there is a better way to deal with this.

Description

Originally, I would submit futures with specific keys from one client and access them from another. Usually this involved a cluster and multiple workers/machines.

Before I would be able to

  1. create a client in one terminal
  2. submit jobs from that terminal with specific keys
  3. connect to the scheduler made by that client in another terminal
  4. re-create the future in the other terminal using Future(key)
  5. access result/status or even cancel future

Now it seems that I am unable to create a Future object directly and the documentation suggests that I should not do so.

Question

Is there another way to deal with this?

Here is hopefully a reproducible example that you can try on both versions.

Example

in one terminal:

from distributed import Client, fire_and_forget

# create client
client = Client(scheduler_port=8786)

def func(x):
    return x + 1

values = list(range(10))
keys = [f"key_{v}" for v in values]

futures = client.map(func, values, key=keys)

# sometimes would use fire and forget if in script rather than in a notebook
fire_and_forget(futures)

in another terminal

from distributed import Client, Future

# connect to previously made client
client = Client('localhost:8786')

# Re-create the futures from another client
futures = [Future(f'key_{v}') for v in range(10)]

# access status / result etc
futures[0].status
futures[0].result()

Thank you for your time. Please let me know if you have any ideas.

Best,
Nick

2

I think you are looking for Variables.

Correct code would be from one terminal:

from distributed import Client, Variable

# create client
client = Client(scheduler_port=8786)

def func(x):
    return x + 1

values = list(range(10))
keys = [f"key_{v}" for v in values]

futures = client.map(func, values, key=keys)

for fut in futures:
    var = Variable(fut.key)
    var.set(fut)

Other terminal:

from distributed import Client, Variable

# connect to previously made client
client = Client('localhost:8786')

variables = [Variable(f'key_{v}') for v in range(10)]
variables[0].get().status
variables[0].get().result()

Didn’t try with fire_and_forget, but it might be more risky!