Specifying default Conda env for Gateway containers

Deploying Dask
,
1

We run a JupyterHub deployed on Kubernetes with the DaskHub Helm chart. It uses a custom Jupyter image that contains a few different conda environments. This image is used for Dask Gateway as well via : DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: ‘{{JUPYTER_IMAGE_SPEC}}’.

I have set DASK_ROOT_CONFIG to be the location of the conda environment to use by default, but it’s not getting picked up and the workers start up with just generic (conda) as the env leading to version mismatches and module not found exceptions when trying to run actual workloads. Also updated is /.bash_profile to load /etc/bash.bashrc which has a command to activate the environment. CONDA_ENV is set, but we do not set ENV_NAME which is how I’ve activated envs in other conda containers.

I thought I had this addressed at one point and it was loading the specified conda env when the cluster was started but now it’s no longer working.

What is the correct way to tell Dask Gateway images the conda environment to use on start?

2

Hi @NickCote,

DASK_ROOT_CONFIG is a folder to put Dask yaml configuration file, I don’t believe it is useed for conda environment.

For Dask Gateway, there is no direct way to give a conda environment to use on start, it will just use the default interpreter of the image. I guess there are several ways you could try to do this:

  • Set appropriate environment variables in the Kubernetes yaml fil under gateway.backend.environment key.
  • Customize you Docker image to set the correct default environment, or use some startup script.
  • Try configuring the worker_cmd in the KubeClusterConfig, trough putting code in the Kubernetes yaml.
3

I had been using the following command in my container image file to activate my conda env

RUN echo ". ${CONDA_DIR}/etc/profile.d/conda.sh ; conda activate ${CONDA_ENV}" > /etc/profile.d/init_conda.sh

This is correct and works. I had another command in my container file for adding color to the cli prompt that was overriding this activation when the dask container launches. I removed that line and can specify the conda environment to activate via the command above.

1 Like