Best Practices for Running Dask Clients with Local Code on a Shared Remote Cluster

Distributed
1

I’m trying to understand the correct way to set up a Dask cluster across remote machines and deploy multiple Dask clients to it, where each client has its own set of Python modules.

For example, I have a project structured like this:

dask-app/
├── poetry.lock
├── pyproject.toml
├── src/
│   └── dask_app/
│       ├── __init__.py
│       ├── main.py              # entry point
│       ├── my_module/ 
│       │   ├── __init__.py
│       │   ├── config.py   
│       │   ├── dask_connector.py 
│       │   └── operations.py    # computation logic
│       └── utils/
│           ├── __init__.py
│           ├── logger.py        
│           └── data_utils.py

and main.py is as follows:

from dask_app.my_module.config import load_config
from dask_app.my_module.dask_connector import get_dask_client
from dask_app.my_module.operations import process_data

def main():

	config = load_config()
    
    # Connect to cluster
    with get_dask_client(config) as client:
        print(f"Connected to cluster: {client.cluster}")
        
        # Perform computations
        result = process_data(client, config)
        print(f"Computation result: {result}")

if __name__ == "__main__":
    main()

What’s the recommended way to deploy such an app to the cluster without restarting the entire Dask cluster every time I update local code?

I’ve come across client.upload_file() but haven’t found a complete example showing how to use it effectively for a project like this, especially when multiple clients with different logic run simultaneously.

2

Hi @doughng, welcome to Dask community!

While it’s possible through upload_file to add some code to a Worker dynamically, I’m not sure if it still is if you want to update a file… Anyway, I’d said that it’s much better to predeploy the environment before starting a Cluster than trying to messing with Python env dynamically.

Could you tell more about your context? What kind of Dask cluster do you use? How each process/node has access to the main code? I’m not sure either of the mutliple clients with different logic.

Generally, Dask is better for ephemeral clusters, one for a given use case and client.