Hello
Receiving error on Worker startup today
Process Dask Worker process (from Nanny)
dask/config.py line 148 -- in update old[k] = v
Type Error: list indices must be integers or slice not str
This same code was working perfectly all last week but only started to have issues
today .
The startup code is
extra_bootstrap = [
"sudo apt-get install software-properties-common",
"sudo add-apt-repository ppa:deadsnakes/ppa -y ",
"sudo apt install ntpdate",
"sudo ntpdate time.nist.gov",
"sudo apt install build-essential",
"sudo apt install gcc",
"sudo apt-get update -y ",
"sudo apt-get install python3.7 -y",
"python -m pip install awscli",
"python -m pip install boto botocore",
"python -m pip install jupyter-server-proxy",
"export DASK_DISTRIBUTED__LOGGING=debug"
]
req_dask_worker_cnt=2
py_packages={"EXTRA_PIP_PACKAGES":"s3fs asyncio lightgbm panda scikit-learn seaborn scipy scikit-learn-intelex matplotlib graphviz joblib nbconvert dpcpp-cpp-rt aiobotocore boto3 aioboto3==11.2.0 "}
req_debug_mode=True
req_auto_shutdown=False
req_docker_args=""
worker_instance_type='r6id.8xlarge' #r means transaction, id -> disk
scheduler_instance_type='m6i.4xlarge' #m means memory, i -> no disk
region_tag="us-east-2"
docker_image_tag="ghcr.io/dask/dask"
cluster = EC2Cluster(
env_vars=py_packages,
debug=req_debug_mode,
filesystem_size=800,
docker_image=docker_image_tag,
security_groups=["sg-0952a59ff138d0edf"],
worker_instance_type=worker_instance_type,
scheduler_instance_type=scheduler_instance_type,
iam_instance_profile={'Arn': 'arn:aws:iam::xxxx:instance-profile/DaskThorProfile'},
n_workers=req_dask_worker_cnt,
security=False,
key_name='awsthoranalytics',
extra_bootstrap=extra_bootstrap,
auto_shutdown=False,
instance_tags={"application":"daskthor"}, # can be set to any name you want
vpc="vpc-0090c55c8d60b5657",
subnet_id='subnet-04f80ecea51538ed8',
availability_zone="us-east-2a",
region='us-east-2',
########################################
enable_detailed_monitoring=True,
# private / public is not relevant to the performance, only influences the security
use_private_ip=False
)
cluster.wait_for_workers(req_dask_worker_cnt)
print("Status - Workers online\n\n ")
Please let me know if you can assist
Thanks