Getting error forbidden 403 when trying to create a cluster in the operator from python but not from yml

Deploying Dask
1

When trying to create a cluster to a running operator in a local minikube env. in ubuntu using version 2023.10.0 with :

 cluster = KubeCluster(
        custom_cluster_spec="./spec.yml",
        namespace="dask-operator",
    )

and i run the file i get the error:

╭─────────────────────── Creating KubeCluster 'example' ───────────────────────╮
│                                                                              │
│   DaskCluster                                                            -   │
│   Scheduler Pod                                                          -   │
│   Scheduler Service                                                      -   │
│   Default Worker Group                                                   -   │
│                                                                              │
│ ⠋                                                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
Traceback (most recent call last):
  File "/root/git/tcc/dasf-core-iac/dask-operator/cluster_spec.py", line 99, in <module>
    cluster = create_dasf_cluster(use_gpu=False)
  File "/root/git/tcc/dasf-core-iac/dask-operator/cluster_spec.py", line 65, in create_dasf_cluster
    cluster = KubeCluster(
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 274, in __init__
    self.sync(self._start)
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 359, in sync
    return sync(
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 426, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 399, in f
    result = yield future
  File "/root/.local/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 314, in _start
    await self._create_cluster()
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 362, in _create_cluster
    raise e
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 353, in _create_cluster
    await cluster.create()
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_objects.py", line 232, in create
    async with self.api.call_api(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_api.py", line 134, in call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://127.0.0.1:50400/apis/kubernetes.dask.org/v1/namespaces/dask-operator/daskclusters'
For more information check: https://httpstatuses.com/403
Exception ignored in atexit callback: <function reap_clusters at 0x7f429c82a3b0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 1025, in reap_clusters
    asyncio.run(_reap_clusters())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 1023, in _reap_clusters
    cluster.close(timeout=10)
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 692, in close
    return self.sync(self._close, timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 359, in sync
    return sync(
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 426, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 399, in f
    result = yield future
  File "/root/.local/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/usr/local/lib/python3.10/dist-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 698, in _close
    cluster = await DaskCluster.get(self.name, namespace=self.namespace)
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_objects.py", line 183, in get
    raise e
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_objects.py", line 177, in get
    resources = await api._get(
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_api.py", line 332, in _get
    async with self._get_kind(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_api.py", line 261, in _get_kind
    async with self.call_api(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/kr8s/_api.py", line 134, in call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://127.0.0.1:50400/apis/kubernetes.dask.org/v1/namespaces/dask-operator/daskclusters'
type or paste code here

From the python file, i printed the context and kubeconfig, both are correct and point towards minikube, but is forbidden. I tried adding RBAC roles to enable the creation, but it didn’t work as well.

When creating the cluster with kubeclt, it works:

kubectl apply -f cluster-spec.yml

I get the same error when I try to connect to the existing cluster created with the operator + pure yml:

cluster = KubeCluster.from_name("example")

I’m sorry if this a minikube issue and not a dask problem

2

Hi @RamonGal,

This looks like some Kubernetes configuration problem, but I really don’t know from where it might come…

cc @jacobtomlinson.

3

Hi @guillaumeeb ,

Thanks for the answer, it indeed was, it sprouted from not having the right RBAC role for the communication between the cluster creator and the Kube operator.

I managed to fix it using the RBAC yml in the documentation as a template, applied to my use case.

In case this helps someone :slight_smile: at :
https://kubernetes.dask.org/en/latest/operator_kubecluster.html?highlight=rbac#role-based-access-control-rbac

1 Like