DASK TLS setup worker issue

Distributed
1

I saw the same issue on my worker. The scheduler looks good. I have set up the config in the yaml file as follows. I already set it to use tls. Any reason that only worker has this error:
I1017 18:46:10.697906 10052059904 main.py:331] RuntimeError: encryption required by Dask configuration, refusing communication from/to ‘tcp://scheduler.join.svc.cluster.local:8786’

  worker:
    replicas: 1
    spec:
      containers:
        - name: worker
          image: {{ .Values.imagePath }}
          args:
            - --mode=worker
            - --name=$(DASK_WORKER_NAME)
            - --worker_memory_limit=12GB
            - --worker_nthreads=1
            - --worker_nworkers=1
          ports:
            - name: http-dashboard
              containerPort: 8788
              protocol: TCP
          env:
            - name: NETRC
              value: ""
            - name: OPENBLAS_NUM_THREADS
              value: "1"
            - name: DASK_DISTRIBUTED__COMM__DEFAULT_SCHEME
              value: "tls"
            - name: DASK_DISTRIBUTED__COMM__REQUIRE_ENCRYPTION
              value: "true"
            - name: DASK_DISTRIBUTED__COMM__TLS__CA_FILE
              value: "/tmp/ca_cert.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__KEY
              value: "/tmp/private_key.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__CERT
              value: "/tmp/cert.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__KEY
              value: "/tmp/private_key.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__CERT
              value: "/tmp/cert.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__KEY
              value: "/tmp/private_key.pem"
            - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__CERT
              value: "/tmp/cert.pem"
            - name: DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT
              value: "60s"
            - name: DASK_DISTRIBUTED__WORKER__USE_FILE_LOCKING
              value: "False"
            - name: DASK_SCHEDULER_ADDRESS
              value: "tls://wei-scheduler.join.svc.cluster.local:8786"
            - name: DASK_TEMPORARY_DIRECTORY
              value: "/tmp"
          volumeMounts:
            - name: tmp
              mountPath: "/tmp"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
      volumes:
        - name: tmp
          emptyDir: {}
2

Hi @will, welcome to Dask community!

It seems you Worker is not using tls when trying to connect to the Scheduler.

What kind of deployment are you using, and what is your complete configuration?

3

I am using helm char to deploy to kubernetes cluster. I set the DASK_SCHEDULER_ADDRESS to use tls in the yaml file.
- name: DASK_SCHEDULER_ADDRESS
value: “tls://wei-scheduler.join.svc.cluster.local:8786”
When I looked at work log, I found dask operator added another tcp config. Do you have any idea on this?
- name: DASK_SCHEDULER_ADDRESS
value: tls://wei-scheduler.join.svc.cluster.local:8786
- name: DASK_TEMPORARY_DIRECTORY
value: /tmp
- name: DASK_WORKER_NAME
value: wei-default-worker-6a9c9e4f94
- name: DASK_SCHEDULER_ADDRESS
value: tcp://wei-scheduler.join.svc.cluster.local:8786

4

I see the dask operator will automatically add a tcp value in the DASK_SCHEDULER_ADDRESS as in the source code. Are there any ways to bypass this setting

5

There is an issue just opened on dask-kubernetes: Reset Dask worker to use TCP even if it was configured to use TLS in yaml file · Issue #836 · dask/dask-kubernetes · GitHub (is that you?).