Hi @Lolomgrofl, welcome to Dask Discourse!
Well, this question is not really specific to Kubernetes, much more to your use case and workflow.
That said, there are a few answers available:
- Dask Best Practices — Dask documentation
- How to configure Dask cluster based on my workload?
- how do we choose --nthreads and --nprocs per worker in dask distributed? - Stack Overflow
Regarding your infrastructure, You’ve got about 5GiB per core, which sound quite normal and adapted to a computing cluster. So there are two big questions: is your workload affected by the GIL, and do you need more than 4/5GiB per process to compute your dataset.
Within Kuernetes, it’s often classical to have only one Worker process per pod, generally with 1 or 2 threads. You could start with pods with 1 process and 2 threads, so requesting 2 vCPU, and about 10 GiB memory. This will give you 30-38 Workers which is a small Dask cluster, but probably enough for what you need. You can also try with 1 process and 1 threads, doubling the number of Workers.
Small pods are easier for Kubernetes to fit on various nodes sizes.