Why does this loop in python runs progressively slower?

Question

In this code, there is a 4-D array of 13x13 images. I would like to save each 13x13 image using matplotlib.pyplot. Here for debugging purposes, I limit the outer loop to 1.

#fts is a numpy array of shape (4000,100,13,13)
no_images = 4000
for m in [1]:  
    for i in range(no_images):
        print i,
        fm = fts[i][m]
        if fm.min() != fm.max():
            fm -= fm.min()
            fm /= fm.max()  #scale to [0,1]
        else:
            print 'unscaled'
        plt.imshow(fmap)
        plt.savefig('m'+str(m)+'_i'+str(i)+'.png')

Saving 4000 images took more than 20 hours. Why is it this slow? If I limit the inner loop to the first 100 images, it takes about 1 minute. So the whole thing should be completed in 40 minutes, not over 20 hours! And I notice it seems to run progressively slower.

Answer 1

What you experience here is a memory leak: you keep creating instances of AxesImage objects (by repetitively calling plt.imshow) to the moment they can't fit into RAM; and then the whole thing begins swapping to disk, which is incredibly slow. To avoid memory leaks, you can either destroy AxesImage instance when you don't need it:

...
image = plt.imshow(fmap)
plt.savefig('m'+str(m)+'_i'+str(i)+'.png')
del(image)

Or, alternatively, you can create only one AxesImage, and then just change the data in it:

...
image = None
for m in [1]:  
    for i in range(no_images):
        ...
        if image is None:
             image = plt.imshow(fmap)
        else:
             image.set_data(fmap)
        ...

Answer 2

I have got the same issue and I tried the above solutions but my dataset is too big for my ram it just collapses after running 20000 images and then I got the answer both plt.close() and del image are not working because they are not clearing total data which is stored they the just adding strain to ram to clear total plt data we need to use plt.figure().clear(), plt.close(), plt.cla(), plt.clf()

This might work for you

#fts is a numpy array of shape (4000,100,13,13)
no_images = 4000
for m in [1]:  
    for i in range(no_images):
        print i,
        fm = fts[i][m]
        if fm.min() != fm.max():
            fm -= fm.min()
            fm /= fm.max()  #scale to [0,1]
        else:
            print 'unscaled'
        plt.imshow(fmap)
        plt.savefig('m'+str(m)+'_i'+str(i)+'.png')
        plt.figure().clear()
        plt.close()
        plt.cla()
        plt.clf()