Instantiate K8s cluster using eksctl

Deploying Dask
1

I am following the dask gateway tutorial (Install on a Kubernetes Cluster — Dask Gateway 2023.1.1 documentation) and would like to create the kubernetes cluster using eksctl from aws. eksctl uses a yaml file to define the cluster structure.

I saw on some older tutorial (Analyze terabyte-scale geospatial datasets with Dask and Jupyter on AWS | AWS Public Sector Blog ) and was able to create the cluster from that config.yaml file (it is for daskhub though, not dask gateway).

My question will be: is there a config file that I can use to instantiate the cluster via eksctl for running dask gateway?

Follow example of the config file that is used in the second tutorial:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: pangeo
  version: "1.21"
  region: us-west-2

availabilityZones: ['us-west-2a','us-west-2b']

iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels: {aws-usage: "cluster-ops"}
    wellKnownPolicies:
      autoScaler: true
    roleName: eksctl-cluster-autoscaler-role
    roleOnly: true
  - metadata:
      name: ebs-csi-controller-sa
      namespace: kube-system
      labels:
        aws-usage: "cluster-ops"
        app.kubernetes.io/name: aws-ebs-csi-driver
    wellKnownPolicies:
      ebsCSIController: true
    roleName: ebs-csi-driver-role
    roleOnly: true

managedNodeGroups:
  - name: main
    minSize: 1
    maxSize: 10
    instanceTypes: ["m5.large", "m5.xlarge"]
    availabilityZones: ["us-west-2a"]
    privateNetworking: true
    volumeEncrypted: true
    iam:
      withAddonPolicies:
        autoScaler: true
        ebs: true
  - name: dask-workers
    minSize: 1
    maxSize: 30
    instanceTypes: ["m5.large", "m5.xlarge", "m5.2xlarge", "m5.4xlarge"]
    availabilityZones: ["us-west-2a"]
    privateNetworking: true
    volumeEncrypted: true
    spot: true
    taints:
    - key: lifecycle
      value: spot
      effect: NoExecute
    iam:
      withAddonPolicies:
        autoScaler: true
        ebs: true

secretsEncryption:
  keyARN: <insert-your-kms-key-arn-here>

Many thanks

2

Well, since daskhub = jupyterhub + dask-gateway, you should be able to use the same configuration file, or almost.

Else, as pointed in Dask Gateway documentation, you should be able to generate one following Kubernetes on Amazon Web Services (AWS) with Elastic Container with Kubernetes (EKS) — Zero to JupyterHub with Kubernetes documentation.

1 Like