How do I set number of workers if my machine specs are like this?

Dask DataFrame
1 
@LAPTOP-DNCQ5AAC:/mnt/d/nom/ml4t/assess_portfolio_2023Sum/assess_portfolio$ l
scpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 5 5600H with Radeon Graphics
    CPU family:          25
    Model:               80
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            6587.43
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
                          pge mca cmov pat pse36 clflush mmx fxsr sse sse
                         2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp l
                         m constant_tsc rep_good nopl tsc_reliable nonsto
                         p_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma
                         cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f1
                         6c rdrand hypervisor lahf_lm cmp_legacy svm cr8_
                         legacy abm sse4a misalignsse 3dnowprefetch osvw
                         topoext perfctr_core ssbd ibrs ibpb stibp vmmcal
                         l fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdse
                         ed adx smap clflushopt clwb sha_ni xsaveopt xsav
                         ec xgetbv1 xsaves clzero xsaveerptr arat npt nri
                         p_save tsc_scale vmcb_clean flushbyasid decodeas
                         sists pausefilter pfthreshold v_vmsave_vmload um
                         ip vaes vpclmulqdq rdpid fsrm
Virtualization features:
  Virtualization:        AMD-V
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   192 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    3 MiB (6 instances)
  L3:                    16 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled vi
                         a prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user
                         pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_F
                         W, STIBP conditional, RSB filling, PBRSB-eIBRS N
                         ot affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
nobu@LAPTOP-DNCQ5AAC:/mnt/d/nomura/ml4t/assess_portfolio_2023Sum/assess_portfolio$ top
top - 18:24:17 up  1:18,  0 users,  load average: 0.54, 0.41, 0.34       Tasks:  59 total,   1 running,  58 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  2.0 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  0.7 si,
%Cpu1  :  1.3 us,  0.0 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu2  :  2.3 us,  0.7 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu5  :  2.3 us,  1.0 sy,  0.0 ni, 96.6 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu6  :  0.7 us,  1.7 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu7  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu8  :  0.7 us,  1.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu9  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu10 :  1.3 us,  1.3 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.0 si,
%Cpu11 :  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,
MiB Mem :   9875.1 total,   1513.3 free,   6162.2 used,   2199.6 buff/ca
MiB Swap:   3072.0 total,   3072.0 free,      0.0 used.   3409.4 avail M

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
 5326 nobu      20   0   11.5g 486180  45232 S   9.3   4.8   1:26.60
 5994 nobu      20   0   11.5g 528968  45244 S   3.3   5.2   1:19.10
 4344 nobu      20   0   21.6g 539408  45260 S   2.0   5.3   1:30.71         1 root      20   0    2324   1708   1600 S   0.0   0.0   0:00.02
    4 root      20   0    2368     68     68 S   0.0   0.0   0:00.18
   11 root      20   0    2340    108      0 S   0.0   0.0   0:00.00
   12 root      20   0    2340    116      0 S   0.0   0.0   0:00.00
   13 nobu      20   0    2888    968    872 S   0.0   0.0   0:00.00
   14 nobu      20   0    2888    968    876 S   0.0   0.0   0:00.00
   19 nobu      20   0    2888    968    876 S   0.0   0.0   0:00.00
   23 nobu      20   0  963208 115748  35048 S   0.0   1.1   0:34.07
   34 nobu      20   0  722640  54684  32504 S   0.0   0.5   0:03.39

I am pulling stock data for my portfolio of 4 stocks over 10 years and comparing its performance to the S&P 500 Index fund. This used to be Pandas assignment at Georgia Tech OMSCS but now it’s online for anyone to try it for the machine learning for trading class.

2

Hi @nyck33,

So, your question is: how should I configure Worker threads vs processes on this machine? Is there one machine per Worker, or as it looks like, it’s a laptop so you are using a LocalCluster?

Well, it’s a question that mainly depends on your Worflow. LocalCluster by default does this (as answered here):

Today that policy is to split all of your logical cores and memory evenly among the square root of the number of cores that you have. So if you have 12 cores then we would create four processes with three threads each.

The general recommandation is to have as many threads * processes as logical cores on your machine. But then depending on your workflow you might want to have more processes or more threads, or one process and only threads or the other way around.