How to avoid 'Too many open files' error when using parallelization within scipy.optimize.differential_evolution?

Question

I am running a python script which uses scipy.optimize.differential_evolution to find optimum parameters for given data samples. I am processing my samples sequentially. The script is running fine, although when I wanted to make use of the parallel computing options implemented in the package, by calling it via:

res = scipy.optimize.differential_evolution( min_fun,
                                             bounds   =     bnds,
                                             updating =    'deferred',
                                             workers  =  120
                                             )

after evaluating res for a few times, it throws an error

File "[...]/miniconda3/envs/remote_fit_environment/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

If I allocate less CPUs, e.g. workers=20 it takes longer, as in more times calling differential_evolution() , until the error occurs.

I am aware that I could raise the limit for open files (1024 by default on that machine), but this seems strange to me. Since the Error origins in the multiprocessing module and is traced back to differential_evolution, I am wondering whether something with the parallelisation implementation within scipy.optimize.differential_evolution might be wrong or "not clean" (although it is much more likely that I am missing something as I am completely new to this whole parallel/multiprocessing thing)?

Any similar experiences or ideas how to solve this?

Answer 1

The root cause is in O/S not having enough "file-descriptors" available for so many process-to-process IPC channels ( as Python IPC as in joblib.parallel() or multiprocessing modules use os.pipe-s for communicating parameters between the "main"-Python-Interpreter process and the (safe)-spawn()-ed / (unsafe, as documented elsewhere why)-fork()-ed sub-ordinate processes, which SciPy re-wraps and re-uses under the calling-signature workers = 120 parameter.

Even if you "increase" the configured amount of O/S filedescriptors ( which intelligent operating systems permit ) for having more os.pipe-instances to work further, the whole intent is not reasonable, as the end-to-end performance will annihilate, as 128-worker processes will have unresolvably self-devastating bottlenecks on CPU-to-RAM physical-RAM-I/O channels on such "over-booked" CPU-to-RAM I/O-traffic density.

Asking more you easily receive less _{( or nothing at all )}

Details matter.