My goal is to process 1000 files at once on a SLURM cluster. I can perform this on 8 nodes each with 32 cores. But then trying with 16 nodes (each with 32 cores) almost in the end of the computation I receive an error message that I have too many open files (Too many open files: ‘/proc/PID/stat’ and…

Hi, The problem comes from the setting of the server from where you’re scheduler is started. See Frequently Asked Questions — Dask.distributed 2022.8.1 documentation . 16 * 32 processes (assuming you still use 1 process for each cores) means 512 workers. So you might reach the limit if you multiply…

Thanks for the reponse. I checked the documentation, but it referred to me to he page, which suggested increasing the limits as a su user, which however I do not have. I am not sure if this is a bug, but while trying to solve the issue today, I figured out that: If I have an open performance repo…

I started a script from a node and it worked. Unfortunately, when I have a performance_report, I get an error of too many file descriptors.

I have the issue that my jobs often start, but from times to times I encounter the too many open files ~ @guillaumeeb . I really cannot argue why?

[image] ikabadzhov: Unfortunately, when I have a performance_report , I get an error of too many file descriptors. Not sure why performance_report does increase that much the number of open files on the Scheduler, but this is not entirely unexpected I guess, Scheduler must be saving a lots of…

Too many open files on SLURMCluster

Distributed

Topic		Replies	Views
Some questions about SLURMCluster dask-jobqueue	5	1411	March 6, 2023
Memory allocation always <= 4GiB for distributed SLURMCluster workers Distributed dask-jobqueue , worker , distributed	8	789	July 12, 2022
SLURMCluster on 64 nodes / Understanding Cluster scale method Distributed distributed	4	350	November 17, 2021
dask_jobqueue.SLURMCluster: multi-threaded workloads and the effect of setting "cores" Distributed dask-jobqueue , distributed	2	315	November 9, 2023
Can not run dask on cluster with LSF Distributed	3	221	January 17, 2024

Too many open files on SLURMCluster

Related topics