"Invalid kube-config file" while using Dask Kubernetes to start a Cluster

Deploying Dask
1

Hi everyone! New dask-deployer here.

I am following the dask Kubernetes documentation in order to deploy dask with Dask Kubernetes.

My MVP code for that is this:

from dask_kubernetes import KubeCluster, make_pod_spec

pod_spec = make_pod_spec(image='daskdev/dask:latest',
                         memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1)

cluster = KubeCluster(pod_spec)

When I run that last line cluster = KubeCluster(pod_spec), I get an error kubernetes_asyncio.config.config_exception.ConfigException: Invalid kube-config file. Expected key current-context in kube-config (I’m happy to post the full slack trace if that’s helpful). Adding a pod manifest manually didn’t resolve the issue, and if I understood the docs, I shoudn’t need to write my own config file if I start the cluster using make_pod_spec().

Does anyone have any ideas what I am doing wrong? Thanks so much!

Notes:
I installed dask-kubernetes with conda install dask-kubernetes -c conda-forge and I’m on version 2022.1.0.

2

Hey there! It sounds like there is something wrong with your Kubernetes config file rather than you doing anything wrong with Dask Kubernetes.

The error states the current context field is missing. Perhaps we are making an assumption that the field should always exist when it doesn’t.

Try running kubectl config set-context NAME. If you’re not sure what the context name is run kubectl config get-contexts.

Then try again :grin:

2 Likes
3

Thanks so much @jacobtomlinson for the reply!

That kubectl line does get me past the ConfigException. I wonder if maybe there are other steps I need to go through to set up kubernetes on my machine, because I get a new config error now:

>>> cluster = KubeCluster(pod_spec)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/core.py", line 474, in __init__
    super().__init__(**self.kwargs)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/distributed/deploy/spec.py", line 260, in __init__
    self.sync(self._start)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/distributed/utils.py", line 309, in sync
    return sync(
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/distributed/utils.py", line 376, in sync
    raise exc.with_traceback(tb)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/distributed/utils.py", line 349, in f
    result = yield future
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/core.py", line 542, in _start
    await ClusterAuth.load_first(self.auth)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/auth.py", line 360, in load_first
    await auth_instance.load()
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/auth.py", line 420, in load
    await self.load_kube_config()
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/auth.py", line 438, in load_kube_config
    loader = self.get_kube_config_loader_for_yaml_file()
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/dask_kubernetes/auth.py", line 424, in get_kube_config_loader_for_yaml_file
    kcfg = KubeConfigMerger(self.config_file)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/kubernetes_asyncio/config/kube_config.py", line 478, in __init__
    self.load_config(path)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/kubernetes_asyncio/config/kube_config.py", line 499, in load_config
    self._merge(item, config[item], path)
  File "/Users/rwegener/.pyenv/versions/miniconda3-4.7.12/envs/dask-kube/lib/python3.10/site-packages/kubernetes_asyncio/config/kube_config.py", line 503, in _merge
    for new_item in add_cfg:
TypeError: 'NoneType' object is not iterable

I’ve never actually used kubernetes before, so I can go through the kubernetes tutorial and see if there is anything about config for my computer.

4

Yeah it definitely looks like an issue with your Kubernetes config file. Normally I would ask folks to share the config so I can have a look but it will contain passwords and secrets, so unless you feel confident you could redact it I wouldn’t advise it.

If you want to DM it to me over on the Dask Slack I’d be happy to have a look.

Otherwise I’m not sure how to more forward here.

2 Likes