Version Mismatching between client and scheduler in Kubecluster

Distributed
1

Hi all and @guillaumeeb,

I am trying to setup single node kubecluster in EC2 instance using minikube and docker. When run the code I am getting warning message which indicates the version mismatching between client and the server.
I just created a conda environement and installed required dask packages within the environment.
I would like to know ,
1 . why I am getting version mismatch?
2. Is there any way to manually setup the versions of the packages (say we want dask=2023.7.1, but scheduler shows latest version that is 2023.8.0)?
3. If I installed everything as latest package, some packages are incompetible with new version (eg: Dask-sql 2023.8.0 requires dask 2023.7.0, in this case again version mismatch problem arrises)
4. To run a kubecluster with dask sql capabilities successfully, what are the packages need to be installed?
5. Which is better : conda install or Pip install for dask-sql

Kindly answer to the above questions, tried a lot to solve the issues but still error arises.
Thanks in advance

2

Hi @Nirajkanth,

If I understand correctly, you are using minikube and Docker for starting the KubeCluster, but your client environment is installed using Conda?

If so, you have one Conda environement, and your KubeCluster uses Docker images, you need to ensure that the software versions in Conda and in Docker are the same.

You need to build the conda environment specifying the versions, or look for Docker images with the versions you want.

If you want to have something consistent, Conda will be the best. You can specify dask-sql version with it and it will ensure the other libraries are compatible. However, you need to also be careful on the Docker image side.

3

Thank you for your valuable time and the suggestions.
It really helps to go deeply in Dask