Failed to create GatewayCluster in EKS setup

Distributed
1

I have an EKS based cluster setup for a proof of concept. After 2 weeks pause; i recreated a cluster, but today with bigger instance types instead of m5.large. My intent want to push the network i/o load to measure a few things. Unfortunately now, from a notebook if I call

cluster = GatewayCluster()

I get an exception

ClientResponseError: 405, message='Method Not Allowed', url=URL('http://proxy-public/services/dask-gateway/api/v1/clusters/')

here are the versions

dask                      2022.8.0           pyhd8ed1ab_1    conda-forge
dask-core                 2022.8.0           pyhd8ed1ab_0    conda-forge
dask-gateway              2022.6.1           pyh8af1aa0_0    conda-forge
dask-labextension         5.3.0              pyhd8ed1ab_1    conda-forge

and my current daskhub.yaml

jupytadminerhub:
  singleuser:
    extraAnnotations:
      iam.amazonaws.com/role: arn:aws:iam::<.....>:role/jupyter-notebook
    image:
      name: pangeo/pangeo-notebook
      tag: "2021.05.04"
    cpu:
      limit: 2
      guarantee: 1
    memory:
      limit: 4G
      guarantee: 2G
    cloudMetadata:
      blockWithIptables: false
    extraEnv:
      DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: '{JUPYTER_IMAGE_SPEC}'
  proxy:
    secretToken: "<secret1.......>"
    https:
      enabled: false
      type: offload
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
  hub:
    config:
      Authenticator:
        admin_users:
          - admin
      DummyAuthenticator:
        password: <secret3.>
      JupyterHub:
        authenticator_class: dummy
    services:
      dask-gateway:
        apiToken: "<secret2.......>"

dask-gateway:
  gateway:
    backend:
      worker:
        extraPodConfig:
          nodeSelector:
            eks.amazonaws.com/capacityType: ON_DEMAND
    extraConfig:
      optionHandler: |
        from dask_gateway_server.options import Options, Integer, Float, String
        def option_handler(options):
            if ":" not in options.image:
                raise ValueError("When specifying an image you must also provide a tag")
            return {
                "worker_cores": options.worker_cores,
                "worker_memory": int(options.worker_memory * 2 ** 30),
                "image": options.image,
            }
        c.Backend.cluster_options = Options(
            Float("worker_cores", default=0.8, min=0.8, max=4.0, label="Worker Cores"),
            Float("worker_memory", default=3.3, min=1, max=8, label="Worker Memory (GiB)"),
            String("image", default="pangeo/base-notebook:2021.05.04", label="Image"),
            handler=option_handler,
        )
    auth:
      jupyterhub:
        apiToken: "<secret2.......>"

What is the cause of the error? 2 Weeks ago I had no problem with it. In the daskhub.yaml I just changed the instance types and the EKS cluster is starting… also I can access notebooks.

2

Today I retry with the original cluster.yaml script, with m5.xlarge instances.
Something which was working well, now is not… the same issue.
Other differences I have noticed. If I open a terminal, now we don’t have aws client anymore.
Clearly somewhere in this stack of dependencies is getting pulled in some changes which breaks the things.