joblib: Worker stopped caused by timeout or memory leak

Question

I am only using the basic joblib functionality:

Parallel(n_jobs=-1)(delayed(function)(arg) for arg in arglist)

I am frequently getting the warning:

UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.

This tells me that one possible cause is a too short worker timeout. Since I did not set a worker timeout and default is None, this cannot be the issue. How do I go about finding a memory leak? Or is there something I can do to avoid this warning? Did some parts not get executed? Or should I just not worry about this?

Answer 1

To fix, increase timeout, I used this:

# Increase timeout (tune this number to suit your use case).
timeout=99999
result_chunks = joblib.Parallel(n_jobs=njobs, timeout=timeout)(joblib.delayed(f_chunk)(i) for i in n_chunks)

Note that this warning is benign; joblib will recover and results are complete and accurate.

See a more detailed answer here.