Use Daskhub in private EKS Cluster with custom CA certificates

[bressanmarcos]

Hey, everyone

I have deployed a Daskhub helm chart in my private EKS cluster. To have https enabled in my private subnets, I have custom SSL certificates.
My users authenticate to Jupyterhub via a self hosted Gitlab instance, available only in the same VPC.

The only problem I am getting right now is that, when I enable HTTPS in JupyterHub, I can’t connect to the proxy-public container (probably from Dask Gateway chart) due to a missing certificate error. I read the docs to find and update every docker image with custom versions, containing the required certificates, but the problem persists.

Does anyone know which Docker images I should update with my root certificate in order to get Daskhub to work properly?

[pavithraes]

@bressanmarcos Thanks for this question! It is similar to Dask Gateway Client SSL configuration, would you mind chiming in there to answer @martindurant’s question? It’ll allow us to help you better!

[bressanmarcos]

FYI, these are my configs in Terraform HCL:

locals {
  config = {
    jupyterhub = {
      debug = {
        enabled = true
      },
      singleuser = {
        image = {
          name = local.image_name,
          tag  = local.image_tag
        },
        cpu = {
          guarantee = 0.1,
          limit     = 4
        },
        memory = {
          guarantee = "1G",
          limit     = "4G"
        },
        extraEnv = {
          DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE = "{JUPYTER_IMAGE_SPEC}"
        }
      },
      scheduling = {
        userScheduler = {
          enabled = true
        },
        podPriority = {
          enabled = true
        },
        userPlaceholder = {
          enabled  = true,
          replicas = 1
        },
        userPods = {
          nodeAffinity = {
            matchNodePurpose = "require"
          }
        },
        corePods = {
          nodeAffinity = {
            matchNodePurpose = "require"
          }
        }
      },
      cull = {
        enabled = true,
        timeout = 3600,
        every   = 300
      },
      hub = {
        image = {
          name = split(":", docker_registry_image.daskhub_hub.name)[0],
          tag  = split(":", docker_registry_image.daskhub_hub.name)[1]
        }
        extraEnv = {
          GITLAB_HOST = var.gitlab_host,
        },
        config = {
          Authenticator = {
            admin_users = var.admin_user_names
          }
        }
      },
      proxy = {
        https = {
          enabled = var.enable_https,
          type    = "manual"
        },
        service = {
          annotations = {
            "service.beta.kubernetes.io/aws-load-balancer-internal"        = "0.0.0.0/0",
            "service.beta.kubernetes.io/aws-load-balancer-security-groups" = join(",", var.load_balancer_sgs)
          },
        }
      }
    },
    "dask-gateway" = {
      gateway = {
        backend = {
          worker = {
            extraPodConfig = {
              nodeAffinity = {
                preferredDuringSchedulingIgnoredDuringExecution = [
                  {
                    weight = 1,
                    preference = {
                      matchExpressions = [
                        {
                          key      = "hub.jupyter.org/node-purpose",
                          operator = "In",
                          values = [
                            "worker"
                          ]
                        }
                      ]
                    }
                  }
                ]
              }
            }
          }
        },
        extraConfig = {
          optionHandler = <<-EOF
            from dask_gateway_server.options import Options, Integer, Float, String
            def option_handler(options):
                if ":" not in options.image:
                    raise ValueError("When specifying an image you must also provide a tag")
                return {
                    "image": options.image,
                }
            c.Backend.cluster_options = Options(
                String("image", default="${local.image_name}:${local.image_tag}", label="Image"),
                handler=option_handler,
            )
          EOF
        }
      }
    }
  }
}

Make notice I rebuilt both jupyterhub.hub.image and jupyterhub.singleuser.image to include my own CA root certificate. Also, I set the load balancer in jupyterhub.proxy.service to be internal-facing.

These are the “secrets-like” parameters:

resource "random_id" "api_token" {
  byte_length = 32
}

locals {
  api_token = sensitive(random_id.api_token.hex)
  secrets = {
    "dask-gateway" = {
      gateway = {
        auth = {
          jupyterhub = {
            apiToken = local.api_token
          }
        }
      }
    },
    jupyterhub = {
      hub = {
        services = {
          "dask-gateway" = {
            apiToken = local.api_token
          }
        },
        extraConfig = {
          "myConfig.py" = <<-EOF
            from oauthenticator.gitlab import GitLabOAuthenticator
            c.JupyterHub.authenticator_class = GitLabOAuthenticator
            c.GitLabOAuthenticator.client_id = '${var.gitlab_oauthenticator.client_id}'
            c.GitLabOAuthenticator.client_secret = '${var.gitlab_oauthenticator.client_secret}'
            c.GitLabOAuthenticator.scope = ['openid', 'profile', 'email', 'read_user']
            c.GitLabOAuthenticator.auto_login = True
            c.GitLabOAuthenticator.oauth_callback_url = '${var.enable_https ? "https" : "http"}://${var.dns_record}/hub/oauth_callback'
        EOF
        }
      },
      proxy = {
        https = {
          manual = {
            key  = var.https_key,
            cert = var.https_cert
          }
        }
      }
    }
  }
}