Hello !
I want to run a AWS Fargate cluster using dask-cloudprovider in a Docker Image to run it in AWS Lambda. I have installed all packages using conda and it works when I’m running it in local. However, I’d like to run it from a AWS Lambda (it is gonna be executed once a day, no need to have a EC2 all the time). I created different Dockerfiles, based on different images.
This last version is one extracted from blog-samples/2021-06-Amazonian-Conda at main · BaysC/blog-samples · GitHub .
- FROM public.ecr.aws/lambda/python:3.8 => I install miniconda with all the dependencies. Same problem, I can’t find my function.
ERROR: “[Errno 30] Read-only file system: ‘/var/task/dask-worker-space’”
FROM public.ecr.aws/lambda/python:3.8
RUN yum update && yum install -y wget && yum clean all
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && sh miniconda.sh -b -p /opt/miniconda
COPY environment.yml /tmp/environment.yml
RUN sed -i -r '/m2w64|vs2015|msys2|win|vc/d' /tmp/environment.yml
RUN /opt/miniconda/bin/conda env create --file /tmp/environment.yml --prefix /opt/conda-env
RUN /opt/conda-env/bin/pip install awslambdaric
RUN mv /var/lang/bin/python3.8 /var/lang/bin/python3.8-orig && ln -sf /opt/conda-env/bin/python /var/lang/bin/python3.8
COPY my_code.py /opt/my-code/my_code.py
ENV AWS_KEY=<KEY> \
AWS_SECRET=<SECRET>
ENV PYTHONPATH "/var/lang/lib/python3.8/site-packages:/opt/my-code"
ENTRYPOINT ["/lambda-entrypoint.sh"]
CMD ["my_code.lambda_handler"]
- So, I changed to to install/copy everything in /tmp since AWS Lambda suggests it. Now the error is: “RequestId: 18… Error: Runtime exited with error: exit status 127” => /var/runtime/bootstrap: line 7: /var/lang/bin/python3.8: No such file or directory
And I have this
FROM public.ecr.aws/lambda/python:3.8
RUN yum update && yum install -y wget && yum clean all
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && sh miniconda.sh -b -p /opt/miniconda
COPY environment.yml /tmp/environment.yml
RUN sed -i -r '/m2w64|vs2015|msys2|win|vc/d' /tmp/environment.yml
RUN /opt/miniconda/bin/conda env create --file /tmp/environment.yml --prefix /tmp/opt/conda-env
RUN /tmp/opt/conda-env/bin/pip install awslambdaric
RUN mv /var/lang/bin/python3.8 /var/lang/bin/python3.8-orig && ln -sf /tmp/opt/conda-env/bin/python /var/lang/bin/python3.8
COPY my_code.py /tmp/opt/my-code/my_code.py
ENV AWS_KEY=<KEY> \
AWS_SECRET=<SECRET>
ENV PYTHONPATH "/var/lang/lib/python3.8/site-packages:/tmp/opt/my-code"
ENTRYPOINT ["/lambda-entrypoint.sh"]
CMD ["my_code.lambda_handler"]
If I run it with a different endpoint just to check the error, and run /var/lang/bin/python3.8 the file is actually there so I don’t understand what’s going on.
PD: The my_code.py file creates a AWS Fargate cluster like:
cluster = FargateCluster(
scheduler_mem=8192, n_workers=12, worker_cpu=256, worker_mem=1024,
aws_access_key_id=AWS_KEY,
aws_secret_access_key=AWS_SECRET,
image="<dependencies_image>",
cloudwatch_logs_group="<cloudwatch_group>"
)
Please let me know if anyone has an idea of what may be going on ! Thanks in advance !
