shutil.rmtree() error when trying to remove NFS-mounted directory

Question

Attempting to execute shutil.rmtree(path) on a directory managed by NFS consistently fails. Below you can see that os.rmdir(path) within shutil.rmtree(path) causes the exception. Is there a more robust way for me to achieve the expected result?

It appears that it removes all of the files, yet a hidden .nfs file remains in the directory for a short amount of time. I'm guessing that the process from which I'm calling rmtree has an open file handle to one of the files inside the directory, which, when deleted, apparently causes NFS to write a new hidden file. That would cause os.rmdir to fail on attempting to remove a non-empty directory.

Traceback (most recent call last):
  File "/home/me/pre3/lib/python3.6/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/me/pre3/lib/python3.6/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty:

NFS details:

$ nfsstat -m
/home/me/nfs from XXX.YYY.ZZZ:/mnt/path/to/nfs
 Flags: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=50,retrans=2,sec=sys,mountaddr=REDACTED,mountvers=3,mountport=832,mountproto=udp,local_lock=none,addr=REDACTED

I'm using Python 3.6.6 on Ubuntu 16.04.

Answer 1

If the python logging module is logging to the target output directory, it will maintain an open file. A workaround is to call logging.shutdown() first, then called shutil.rmtree(path). This is not a general answer to the broader question, however.

Answer 2

You could try defining an error handler function to be passed to the onerror arg for shutil.rmtree: https://docs.python.org/3/library/shutil.html#shutil.rmtree

def handle_rmtree_err(function, path, excinfo):
    ... 

shutil.rmtree(my_path, onerror=handle_rmtree_err)

There are all sorts of reasons why a process may be holding onto a file, so I can't tell you what the error handler should do exactly.

If you haven't figured out what is holding onto the file, try $ lsof | grep .nfsXXXX.

If all else fails you could time.sleep(secs) and retry shutil.rmtree.